System and method for training artificial intelligence systems using a sima based processor

ABSTRACT

A reinforcement learning processor specifically configured to train reinforcement learning agents in the AI systems by the way of implementing an application-specific instruction set is disclosed. The application-specific instruction set incorporates ‘Single Instruction Multiple Agents (SIMA)’ instructions. SIMA type instructions are specifically designed to be implemented simultaneously on a plurality of reinforcement learning agents which interact with corresponding reinforcement learning environments. The SIMA type instructions are specifically configured to receive either a reinforcement learning agent ID or a reinforcement learning environment ID as the operand. The reinforcement learning processor is designed for parallelism in reinforcement learning operations. The reinforcement learning processor executing of a plurality of threads associated with an operation or task in parallel.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part which claims priority fromU.S. patent application Ser. No. 15/455,126, filed on Mar. 9, 2017,entitled “PROCESSOR FOR IMPLEMENTING REINFORCEMENT LEARNING OPERATIONS”.

BACKGROUND Technical Field

The present disclosure relates to the field of reinforcement learning.Particularly, the present disclosure relates to a processor specificallyprogrammed for training Artificial Intelligence systems, and to anapplication-domain specific instruction set (ASI) comprisinginstructions specifically designed for training reinforcement learningagents.

Description of the Related Art

Artificial Intelligence (AI) aims to make a computer/computer-controlledrobot/computer implemented software program mimic the thought process ofa human brain. Artificial Intelligence is utilized in various computerimplemented applications including gaming, natural language processing,creation and implementation of expert systems, creation andimplementation of vision systems, speech recognition, handwritingrecognition, and robotics. A computer/computer controlled robot/computerimplemented software program achieves or implements ArtificialIntelligence through iterative learning, reasoning, perception,problem-solving and linguistic intelligence.

Machine learning is a branch of artificial intelligence that providescomputers the ability to learn without necessitating explicit functionalprogramming. Machine learning, emphasizes on the development of(artificially intelligent) learning agents that could tweak theiractions and states dynamically and appropriately when exposed to a newset of data. Reinforcement learning is a type of machine learning wherea reinforcement learning agent learns by utilizing the feedback receivedfrom a surrounding environment in each entered state. The reinforcementlearning agent traverses from one state to another by the way ofperforming an appropriate action at every state, thereby receiving anobservation/feedback and a reward from the environment. The objective ofa Reinforcement Learning (RL) system is to maximize the reinforcementlearning agent's total rewards in an unknown environment through alearning process that warrants the reinforcement learning agent totraverse between multiple states while receiving feedback and reward atevery state, in response to an action performed at every state.

Further, essential elements of a reinforcement learning system include a‘policy’, ‘reward functions’, action-value functions’ and ‘state-valuefunctions’. Typically, a ‘policy’ is defined as a framework forinteraction between the reinforcement learning agent and a correspondingreinforcement learning environment. Typically, the actions undertaken bythe reinforcement learning agent and the states traversed by thereinforcement learning agent during an interaction with a reinforcementlearning environment are governed by the policy. When an action isundertaken, the reinforcement learning agent moves within theenvironment from one state to another and the quality of a state-actioncombination defines an action-value function. The action-value function(Q_(s)) determines expected utility of a (selected) action. The rewardfunction is representative of the rewards received by the reinforcementlearning agent at every state in response to performing a predeterminedaction. Even though rewards are provided directly by the environmentafter the reinforcement learning agent performs specific actions, the‘rewards’ are estimated and re-estimated (approximated/forecasted) fromthe sequences of observations a reinforcement learning agent makes overits entire lifetime. Thus, a reinforcement learning, algorithm aims toestimate state-value function and an action-value function that helpsapproximate/forecast the maximum possible reward to the reinforcementlearning agent.

Q-learning is one of the techniques employed to perform reinforcementlearning. In Q-learning, the reinforcement learning agent attempts tolearn an optimal policy based on the historic information correspondingto the interaction between the reinforcement learning agent andreinforcement learning environment. The reinforcement learning agentlearns to carry out actions in the reinforcement learning environment tomaximize the rewards achieved or to minimize the costs incurred.Q-learning estimates the action-value function that further provides theexpected utility of performing a given action in a given state andfollowing the optimal policy thereafter. Thus, by finding the optimalpolicy, the agents can perform actions to achieve maximum rewards.

Existing methods disclose the use of neural networks (by thereinforcement learning agents) to determine the action to be performedin response to the observation/feedback. Neural networks are machinelearning models that employ one or more layers of nonlinear tints topredict an output for a received input. Each layer of the networkgenerates an output from a received input in accordance with currentvalues of a respective set of parameters. However, existing methods donot disclose processor architectures specifically configured to performreinforcement learning operations. Furthermore, existing methods thatpromote the use of neural networks do not support reward functionapproximation.

To implement the function of deep reinforcement learning, and AIapplication, existing systems typically use GPUs. GPUs typicallyincorporate Single Instruction Multiple Data (SIMD) architecture toexecute reinforcement learning operations. In SIMD, all the GPUs sharethe same instruction but perform operations on different data elements.However, the GPUs require a large amount of processing time to extractactionable data. Further, GPUs are unsuitable for sequentialdecision-making tasks and are hindered by a lack of efficiency as far asprocessing the memory access of reinforcement learning tasks isconcerned.

Further, the existing systems do not provide parallelism forreinforcement learning operations. Typically, it consumes about fifteendays to train a full RL agent on the GPU. Reinforcement learninginherently controls data distribution, unlike deep learning, where datamust be fed into network. However, the Processors such as GPUs and CPUsincur huge hierarchical overhead of memories. This is because GPUs andCPUs have memory hierarchy in terms of Cache and main memory. Also, thepower consumed in such training process is very high and enormous. Eachmachine consumes more than 600 W of power. Hence, many such GPUs or CPUsare required to perform a distributed training process. However, it isrequired to reduce the training time by a processor architecture withmulti agents and distributed training.

Therefore, in order to overcome the drawbacks discussed hitherto, thereis a need for a processor architecture specifically designed forimplementing reinforcement learning operations/tasks. Further, there isa need for a processor architecture that renders rich actionable datafor effective and efficient implementation of reinforcement learningoperations. Yet further, there is a need for a processor architecturethat incorporates an application-domain specific instruction set, amemory architecture and a multi-core processor specifically designed forperforming reinforcement learning tasks/operations. Furthermore, thereis a need for a processor architecture that incorporates SIMAinstruction set with multiple agents for distributed training. Stillthere is a need for a massive parallelism for reinforcement learningwhile training the reinforcement learning using SIMA based processor(RAP processor).

The abovementioned shortcomings, disadvantages and problems areaddressed herein, which will be understood by reading and studying thefollowing specification.

Objects

A primary object of the present disclosure is to provide a processorarchitecture that enables extraction and collection of rich actionabledata best suited for reinforcement learning operations.

Another object of the present disclosure is to provide a processorarchitecture (processor) that is tailor made for effective and efficientimplementation of reinforcement learning tasks/operations.

Yet another object of the present disclosure is to provide a processorarchitecture incorporating instruction set based on Single InstructionMultiple Agents (SIMA) which is specifically designed for executingtasks/operations pertinent to reinforcement learning in parallel.

Yet another object of the present disclosure is to provide anapplication domain specific instruction set that is simultaneouslyexecuted across multiple reinforcement learning agents and reinforcementlearning environments (Single Instruction Multiple Agents (SlMA)).

Yet another object of the present disclosure is to provide a processorarchitecture designed for parallelism in reinforcement learningoperations.

Yet another object of the present disclosure is to provide a processorarchitecture designed to reduce training times by a processorarchitecture with multi agents and distributed training

Yet another object of the present disclosure is to provide anapplication domain specific instruction set and a processor architectureoptimized for training the reinforcement learning agents.

Yet another object of the present disclosure is to provide SIMA (SingleInstruction Multiple Agents) ISA based processor, in which operand is anenvironment ID or an agent ID, and the operand is scalar or vector.

Yet another object of the present disclosure is to provide autonomoussystem and server based applications, in which the multiple agents aregiven as operand to the SIMA instructions, for speeding up training inthe applications like gaming.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems using SIMA (Single Instruction Multiple Agents)ISA based processor, to support both the reply memory and as well asparallel actor-critic agents, thereby supporting a number of on policylearning methods.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems using SIMA (Single Instruction Multiple Agents)based processor, to provide a series of instructions from the hostprocessor to multi-core, multi-threaded RLP.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems to provide each agent with different r, v, q, gand a threads, thereby providing separate thread for each one ofreinforcement learning operations, to effectively pipeline a number ofparallel agents, to decrease training time by order of magnitudecompared to Advantage Actor Critic (A3C) algorithm where entire agent ismapped to a single thread.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems to enable multiple agents to explore differentpart of the environment thereby using lesser resource than single DQN,and reducing communication overhead.

Yet another of the present disclosure is to provide a training systemfor AI systems to use fine grain threads of compared to single thread onentire agent, to enable supporting large number of agents in ahierarchical task.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems using SIMA to provide same instructions tomultiple agents to reduce code density and create highly pipelinedoperations with large parallelism in training.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems, to group similar of different agents into onethread blocks of operations with SIMA instructions.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems to create the agents (actor plus critic) and mapthe threads, instead of mapping entire of algorithm into single thread.

Yet another object of the present disclosure is to provide a trainingsystem for AI systems, map each of the operations into a micro threadand to pipeline these micro threads, instead of napping entire agentinto a single thread, thereby increasing the throughput of theprocessor.

Yet another object of the present disclosure is to provide a trainingsystem for performing thread mapping to Single-step Q learning, Sarsa, nstep Q learning etc.,

Yet another object of the present disclosure is to provide anapplication domain specific instruction set and a processor architecturethat creates an effective balance between exploration and exploitationof a reinforcement learning environment.

Yet another object of the present disclosure is to provide anapplication domain specific instruction set and a processor architecturethat enable parallel learning and effective sharing of learning, amongsta plurality of reinforcement learning agents.

Yet another object of the present disclosure is to provide a processorarchitecture that necessitates fewer clock cycles in comparison to theconventional CPU/GPU, to implement reinforcement learningoperations/tasks.

Yet another object of the present disclosure is to provide anapplication domain specific instruction set and a processor architecturethat renders comparatively larger levels of abstraction, during theimplementation of reinforcement learning operations/tasks.

These and other objects and advantages of the present invention willbecome readily apparent from the following detailed description taken inconjunction with the accompanying drawings.

SUMMARY

In order to overcome the drawbacks discussed hitherto, the presentdisclosure envisages processor architecture specifically designed toimplement reinforcement learning operations. The processor architectureprovides rich actionable data for scientific computing, cloud computing,robots, and IOT computing inter-ilia. The processor architectureincludes a first processor (host processor), a first memory module(IRAM), a Complex Instruction fetch and decode (CISFD) unit, a secondprocessor (Reinforcement learning processor), and a second memorymodule. In accordance with the disclosure, the processor architecture isconfigured for parallelism m reinforcement learning operations. Thearchitecture is configured to train reinforcement learning agents usingsingle instruction multiple agents (SIMA) based instructions to cause aplurality of agents following different policies execute in copies ofsame environments. Further, the best of all agents is selected andmerged in a synchronous or asynchronous manner. Also, the processorarchitecture is designed to reduce training times by a processorarchitecture with multi agents and distributed training.

According to an embodiment herein, a computer-implemented system isprovided for training artificial intelligence based systems. The systemcomprises a first processor that is configured to create at least onereinforcement learning agent and at least one correspondingreinforcement learning environment. The first processor is furtherconfigured to assign a reinforcement learning agent ID to thereinforcement learning agent, and a reinforcement learning environmentID to the reinforcement learning environment.

A first memory module is communicably coupled to the first processor.The first memory module is configured to store an application-domainspecific instruction set (ASI). The application-domain specificinstruction set comprises instructions for implementing thepredetermined reinforcement learning operations. Each of theinstructions incorporates at least one of the reinforcement learningagent ID and the reinforcement learning environment ID is treated as anoperand.

A complex instruction fetch and decode (CISFD) unit is communicablycoupled to the memory module. The CISFD unit is configured to decode atleast one of the instructions, and generate a plurality of predeterminedthreads corresponding to decoded instruction. The CISFD unit is stillfurther configured to embed at least one of the reinforcement learningagent ID and reinforcement learning environment ID associated with thedecoded instruction into the predetermined threads.

A second processor is configured to execute the predeterminedreinforcement learning operations. The second processor comprises aplurality of cores. The second processor is configured to process eachof the predetermined threads in parallel using the plurality of cores,and apply a corresponding instruction onto at least one of areinforcement learning agent. A reinforcement learning environment isidentified by the reinforcement learning agent ID and reinforcementlearning environment ID is embedded in the predetermined threads.

A second memory module is configured to cooperate with the secondprocessor. The second memory module is a gradient memory. A plurality ofgradients in the gradient memory is updated from each one of a pluralityof agents working in parallel.

The second processor is further configured to train the reinforcementlearning agents using application-domain specific instruction set (ASI),and wherein the application-domain specific instruction set are singleinstruction multiple agents (SIMA) based instructions. A plurality ofagents is configured to work on a copy of same environment and work onmutually different policies in parallel. The plurality of agents isselected and merged in a synchronous or asynchronous manner.

According to an embodiment herein, the second processor is configured tocreate environment vectors corresponding to the reinforcement learningenvironment; create agent vectors corresponding to the plurality ofreinforcement learning agents; associate the plurality of reinforcementlearning agents to at least one reinforcement learning environment, andwherein the reinforcement learning agents follows a differentexploration policies as compared with the reinforcement learningenvironment; initiate a training on the reinforcement learning agentrepresented by the reinforcement learning agent ID by using explorationinstruction; generate a plurality of thread vectors and micro threads bythe CISFD in response to initiation of training, and wherein theplurality of thread vectors comprises q-thread vector, a-thread vector,v-thread vector, g-thread vector and r thread vector, and wherein eachoperation is mapped into micro thread; map a plurality of reinforcementlearning operations to the plurality of thread vectors, and wherein eachthread vector corresponds to a plurality of reinforcement learningagents; update a gradient vector corresponding to at least one of anagent actor, and an agent critic, and wherein the agent actorcorresponds to the q-thread vector and the agent critic corresponds tothe v-thread vector; update an action vector from the agent actorcorresponding to the a-thread; determine a gradient parameter of policydθ and a gradient parameter of value function dθv vector correspondingto the g-thread vector; and update a policy parameter θ and a parameterof value function θv from determined gradient parameters dθ and dθv in agradient memory of the second memory module.

According to an embodiment herein, the plurality of thread vectorsgenerated by the CISFD are indicative of the interaction between thereinforcement learning agent and the reinforcement learning environment.The plurality of thread vectors are executed for a predetermined timefor each reinforcement learning agent.

According to an embodiment herein, the second processor is furtherconfigured to: trigger the reinforcement learning agent to interact withthe reinforcement learning environment and enable the reinforcementlearning agent to explore the reinforcement learning environment todeduce relevant learning therefrom; apply deduced relevant learning ontothe reinforcement learning environment; trigger the reinforcementlearning agent to exploit gathered learnings by further interacting withthe reinforcement learning environment; construct at least one of aMarkov Decision Process (MDP) and Semi-Markov Decision Process (SMDP)based on an interaction between the reinforcement learning agent and thereinforcement learning environment; selectively update the MDP and SMDP,based on learning achieved from the interaction between thereinforcement learning agent and the reinforcement learning environment;selectively create a backup of the MDP representative of the interactionbetween the reinforcement learning agent and the reinforcement learningenvironment; initiate a q-learning procedure on the reinforcementlearning agent; associate a reward function with the reinforcementlearning agent; enable the reinforcement learning agent to interact withthe reinforcement learning environment for a predetermined time period;read a context corresponding to the reinforcement learning agent, andread the context corresponding to the reinforcement learningenvironment, and store the read context; initiate a predetermineddeep-learning operation on the reinforcement learning agent; determinethe optimal Q-value corresponding to a current state of thereinforcement learning agent; implement generalized policy iteration(GPI); implement on-policy and off-policy learning methods on thereinforcement learning agent; approximate a state-value function for thecurrent state of the reinforcement learning agent, and determine anoptimal state-value function for the current state of the reinforcementlearning agent; and fetch Q-values, state-value functions and rewardvalues corresponding to the reinforcement learning agent, from thecorresponding memory partitions of the second memory module.

According to an embodiment herein, the second processor is furtherconfigured to feed training data into the neural network through theneural network data path, and initiate training of the neural network;download weights corresponding to the neural network; communicate with apredetermined deep neural network; selectively clear the weightsassigned to the neural network; perform value-function approximationusing the deep neural network, for the reinforcement learning agentrepresented by the reinforcement learning agent ID; and train the deepneural network using a predetermined learning context.

According to an embodiment herein, the instructions are configured to besimultaneously implemented on a plurality of reinforcement learningagents and a plurality of reinforcement learning environments. The CISFDunit is further configured to create the predetermined threadscorresponding to each of the plurality of reinforcement learning agentsand plurality of reinforcement learning environments. The CISFD unitstill further configured to group the predetermined threads into athread block.

According to an embodiment herein, the second memory module ispartitioned into a gradient memory to store Global parameters θ, θvobtained from the training.

According to an embodiment herein, a method for training artificialintelligence based systems, is provided. The method comprising followingcomputer-implemented steps: at least one reinforcement learning agentand at least one corresponding reinforcement learning environment arecreated using a first processor. A reinforcement learning agent ID isassigned to the reinforcement learning agent. A reinforcement learningenvironment ID is assigned to the reinforcement learning environment. Anapplication-domain specific instruction set (ASI) is created. Theapplication-domain specific instruction set comprises instructions forimplementing the predetermined reinforcement learning operations. Eachinstruction is created by incorporating at least one of the realreinforcement learning agent ID and the reinforcement learningenvironment ID therein, as an operand. The application-domain specificinstruction set is stored in a first memory module. At least one of theinstructions is selectively fetched and decoding using a complexinstruction fetch and decode (CISFD) unit. A plurality of predeterminedthreads is generated corresponding to the decoded instruction. At leastone of the reinforcement learning agent ID and reinforcement learningenvironment ID associated with the decoded instruction is embedded intothe predetermined threads. Each one of the predetermined dreads isexecuted in parallel using a second processor. The correspondinginstruction is applied onto at least one of a reinforcement learningagent and reinforcement learning environment identified respectively bythe reinforcement learning agent ID and reinforcement learningenvironment ID embedded in the predetermined threads. The reinforcementlearning agents are trained using the second processor by executingapplication-domain specific instruction set (ASI). Theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions.

According to an embodiment herein, the step of creating anapplication-domain specific instruction set (ASI), and incorporating theinstructions into the ASI for creating environment vectors correspondingto the reinforcement learning environment; creating agent vectorscorresponding to the plurality of reinforcement learning agents;associating the plurality of reinforcement learning agents to at leastone reinforcement learning environment, and wherein the reinforcementlearning agents follows a different exploration policies as comparedwith the reinforcement learning environment; initiating a training onthe reinforcement learning agent represented by the reinforcementlearning agent ID by using exploration instruction; generating aplurality of thread vectors and micro threads by the CISFD in responseto initiation of training, and wherein the plurality of thread vectorscomprises q-thread vector, a-thread vector, v-thread vector, g-threadvector and r thread vector; mapping a plurality of reinforcementlearning operations to the plurality of thread vectors, and wherein eachthread vector corresponds to a plurality of reinforcement learningagents; updating a gradient vector corresponding to at least one of anagent actor, and an agent critic, and wherein the agent actorcorresponds to the q-thread vector and the agent critic corresponds tothe v-thread vector; updating an action vector from the agent actorcorresponding to the a-thread; determining a gradient parameter ofpolicy dθ and gradient parameter of value function dθv vectorcorresponding to the g-thread vector; and updating policy parameter θand parameter of value function θv from determined gradient parametersdθ and dθv in a gradient memory of the second memory module.

According to an embodiment herein, the plurality of thread vectorsgenerated by the CISFD are indicative of the interaction between thereinforcement learning agent and the reinforcement learning environment.The plurality of thread vectors are executed for a predetermined timefor each reinforcement learning agent.

According to an embodiment herein, the step of creating anapplication-domain specific instruction set (ASI), further includes thestep of incorporating instructions into the ASI, for triggering thereinforcement learning agent to interact with the reinforcement learningenvironment and enabling the reinforcement learning agent to explore thereinforcement learning environment to deduce relevant learningtherefrom; apply deduced relevant learning onto the reinforcementlearning environment; trigger the reinforcement learning agent toexploit gathered learning by further interacting with the reinforcementlearning environment; construct at least one of a Markov DecisionProcess (MDP) and Semi-Markov Decision Process (SMDP) based oninteraction between the reinforcement learning agent and thereinforcement learning environment; selectively update the MDP and SMDP,based on learning achieved from the interaction between thereinforcement learning agent and the reinforcement learning environment;selectively create a backup of the MDP representative of the interactionbetween the reinforcement learning agent and the reinforcement learningenvironment; initiate a q-learning procedure on the reinforcementlearning agent; associate a reward function with the reinforcementlearning agent; perform generalized policy iteration (GPI); implementon-policy and off-policy learning methods on the reinforcement learningagent; enable the reinforcement learning agent to interact with thereinforcement learning environment for a predetermined time period; reada context corresponding to the reinforcement learning agent, and readthe context corresponding to the reinforcement learning environment, andstore the read context; initiate a predetermined deep-learning operationon the reinforcement learning agent represented by the reinforcementlearning agent ID; determine the optimal Q-value corresponding to acurrent state of the reinforcement learning agent; fetch the Q-values,state-value functions, actions and reward values corresponding to thereinforcement learning agent, from the corresponding memory partitionsof the second memory module; approximate a state-value function for thecurrent state of the reinforcement learning agent, and determine anoptimal state-value function for the current state of the reinforcementlearning agent.

According to an embodiment herein, the step of executing each of thepredetermined threads in parallel using a second processor, and applyingthe corresponding instruction onto at least one of a reinforcementlearning agent and reinforcement learning environment identifiedrespectively by the reinforcement learning agent ID and reinforcementlearning environment ID embedded in the predetermined threads, anddetermining an action, a state-value function, Q-value and a rewardvalue, based on execution of the instruction corresponding to thepredetermined threads, further includes the following steps. TheQ-values are determined based on the execution of a q-thread. Theactions to be performed by the reinforcement learning processor aredetermined based on the execution of a-thread. The state-value functionsare determined based on the execution of a v-thread. The reward valuesare determined based on the execution of an r-thread.

According to an embodiment herein, the step of determining an actionbased on the execution of a-thread, further includes the step ofdetermining the action performed by the reinforcement learning agent ina predetermined state, and under a predetermined policy.

According to an embodiment herein, the step of determining thestate-value functions based on the execution of a v-thread, furtherincludes the step of determining a value associated with each state ofthe reinforcement learning agent, and representing the value as astate-value function.

According to an embodiment herein, the step of determining the rewardvalues based on the execution of an r-thread, further includes the stepof determining rewards obtained by the reinforcement learning agent forperforming a predetermined action in a predetermined state, within thereinforcement learning environment, and storing the rewards as rewardvalues.

According to an embodiment herein, a computer readable non-transitorystorage medium is stored with computer readable instructions that areexecuted by a computer processor, to cause the computer processor tocreate at least one reinforcement learning agent and at least onecorresponding reinforcement learning environment, and assign areinforcement learning agent ID to the reinforcement learning agent, andassign a reinforcement learning environment ID to the reinforcementlearning environment; create an application-domain specific instructionset (ASI), the application-domain specific instruction set comprisinginstructions for implementing the predetermined reinforcement learningoperations; create each of the instructions by incorporating at leastone of the reinforcement learning agent ID and the reinforcementlearning environment ID therein, as an operand; configure each of theinstructions to be simultaneously implemented on a plurality ofreinforcement learning agents and a plurality of reinforcement learningenvironments; selectively fetch at least one of the instructions byusing a complex instruction fetch and decode (CISFD) unit, and decodethe instruction, and generate a plurality of predetermined threadscorresponding to decoded instruction; embed into the predeterminedthreads, at least one of the reinforcement learning agent ID andreinforcement learning environment ID associated with correspondinginstruction; process each of the predetermined threads in parallel, andapply the corresponding instruction onto at least one of a reinforcementlearning agent and reinforcement learning environment identified by thereinforcement learning agent ID and reinforcement learning environmentID embedded in the predetermined threads; train the reinforcementlearning agents using the second processor by executingapplication-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions.

According to an embodiment herein, the computer readable instructions,when executed by the computer processor, further cause the computerprocessor to create environment vectors corresponding to thereinforcement learning environment; create agent vectors correspondingto the plurality of reinforcement learning agents; associate theplurality of reinforcement learning agents to at least one reinforcementlearning environment, and wherein the reinforcement learning agentsfollows a different exploration policies as compared with thereinforcement learning environment; initiate a training on thereinforcement learning agent represented by the reinforcement learningagent ID by using exploration instruction; generate a plurality ofthread vectors and micro threads by the CISFD in response to initiationof training, and wherein the plurality of thread vectors comprisesq-thread vector, a-thread vector, v-thread vector, g-thread vector and rthread vector; map a plurality of reinforcement learning operations tothe plurality of thread vectors, and wherein each thread vectorcorresponds to a plurality of reinforcement learning agents; update agradient vector corresponding to at least one of an agent actor, and anagent critic, and wherein the agent actor corresponds to the q-threadvector and the agent critic corresponds to the v-thread vector; updatean action vector from the agent actor corresponding to the a-thread;determine a gradient parameter of policy dθ and a gradient parameter ofvalue function dθv vector corresponding to the g-thread vector; andupdate a policy parameter θ and a parameter of value function θv fromdetermined gradient parameters dθ and dθv in a gradient memory of thesecond memory module.

According to an embodiment herein, the plurality of thread vectorsgenerated by the CISFD are indicative of the interaction between thereinforcement learning agent and the reinforcement learning environment,the plurality of thread vectors are executed for a predetermined timefor each reinforcement learning agent.

The host processor is configured to create at least one reinforcementlearning agent and at least one reinforcement learning environment.Further, the host processor assigns an agent ID and environment ID tothe reinforcement learning agent and the reinforcement learningenvironment respectively.

In accordance with the present disclosure, the IRAM is coupled to thereinforcement learning processor and is configured to store anapplication-domain specific instruction set (ASI). Theapplication-domain specific instruction set (ASI) includes instructionsoptimized for performing reinforcement learning operations. Theinstructions incorporate at least one of the reinforcement learningagent ID and the reinforcement learning environment ID as an operand.The CISFD unit is configured to fetch upto ‘N’ instructionssimultaneously for decoding. The CISFD unit generates a plurality ofthreads, for example, an r-thread, a v-thread, a q-thread, and ana-thread corresponding to a decoded instruction. Each of the threads areembedded with either the reinforcement learning agent ID orreinforcement learning environment ID (depending upon the correspondinginstruction). The threads corresponding to the decoded instruction aretransmitted to the reinforcement learning processor. The threads areexecuted in parallel using a plurality of processing cores of thereinforcement learning processor. In an example, if ‘N’ is the number ofprocessor cores, then ‘N’ instructions are fetched by CISFD forsimultaneous execution.

In accordance with the present disclosure, the reinforcement learningprocessor is a multi-core processor. Each processor core of thereinforcement learning processor includes a plurality of executionunits. Further, each execution unit includes a fetch/decode unit,dispatch/collect unit, and a plurality of registers for storing learningcontext and inferencing context corresponding to a reinforcementlearning agent. The fetch/decode unit is configured to fetch the threadscorresponding to the decoded instruction. Subsequently, the executionunit performs ALU operations corresponding to the threads, on theregisters storing the learning context and the inferencing context. Theresults (of the execution of threads) are generated based on thelearning context and inferencing context stored in the registers.Subsequently, the results are transmitted to the collect/dispatch unitwhich stores the results (of the execution of threads) in predeterminedpartitions of a second memory module.

In accordance with the present disclosure, subsequent to the executionof threads corresponding to a decoded instruction, an action (performedby the reinforcement learning agent at every state), at least onestate-value function, at least one Q-value, and at least one rewardfunction are determined. The action, state-value function, Q-value, andreward function thus generated represent the interaction between thereinforcement learning agent and the corresponding reinforcementlearning environment (either of which was specified as an operand in theinstruction executed by the reinforcement learning processor).

In accordance with the present disclosure, the second memory module ispartitioned into an ‘a-memory module’, a ‘v-memory module’, a ‘q-memorymodule’, and an ‘r-memory module’. The ‘a-memory module’ storesinformation corresponding to the action(s) performed by thereinforcement learning agent at every state, during an interaction withthe reinforcement learning environment. Further, the ‘v-memory module’stores the ‘state-value functions’ which represent the value associatedwith the reinforcement learning agent at every state thereof. Further,the ‘q-memory module’ stores ‘Q-values’ which are generated using astate-action function indicative of the action(s) performed by thereinforcement learning agent at every corresponding state. Further, the‘r-memory module’ stores the information corresponding to the ‘rewards’(also referred to as reward values) obtained by the reinforcementlearning agent in return for performing a specific action while being ina specific state.

In accordance with the present disclosure, the reinforcement learningprocessor uses neural network data paths to communicate with a neuralnetwork which in turn uses the actions, state-value functions, Q-valuesand reward values generated by the reinforcement learning processor toapproximate an optimal state-value function as well as an optimal rewardfunction. Further, the neural network also programs the reinforcementlearning agent with a specific reward function, which dictates theactions to be performed by the reinforcement learning agent to obtainthe maximum possible reward.

These and other aspects of the embodiments herein will be betterappreciated and understood when considered in conjunction with thefollowing description and the accompanying drawings. It should beunderstood, however, that the following descriptions, while indicatingthe preferred embodiments and numerous specific details thereof, aregiven by way of illustration and not of limitation. Many changes andmodifications may be made within the scope of the embodiments hereinwithout departing from the spirit thereof, and the embodiments hereininclude all such modifications.

Although the specific features of the embodiments herein are shown insome drawings and not in others. This is done for convenience only aseach feature may be combined with any or all of the other features inaccordance with the embodiments herein.

BREIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

The other objects, features and advantages will be apparent to thoseskilled in the art from the following description and the accompanyingdrawings. In the accompanying drawings, like numerals are used torepresent/designate the same components previously described.

FIG. 1 illustrates a block-diagram of a system for implementingpredetermined reinforcement learning operations, in accordance with thepresent disclosure.

FIG. 1A illustrates a block diagram of a processor core of thereinforcement learning processor, in accordance with the presentdisclosure.

FIG. 1B illustrates a block diagram of an execution unit of theprocessor core, in accordance with the present disclosure.

FIG. 1C illustrates a block diagram indicating the format of ‘threadID’,in accordance with the present disclosure.

FIG. 1D illustrates a block diagram indicating the format for addressingthe memory partitions of the second memory module, in accordance withthe present disclosure.

FIG. 1E illustrates a block diagram indicating the memory bankscorresponding to the v-memory module, in accordance with the presentdisclosure.

FIG. 1F illustrates a block diagram indicating the memory bankscorresponding to the q-memory module, in accordance with the presentdisclosure.

FIG. 1G illustrates a block diagram indicating the memory bankscorresponding to the r-memory module, in accordance with the presentdisclosure.

FIG. 2A is a block diagram illustrating the agent context correspondingto the reinforcement learning agent, in accordance with the presentdisclosure.

FIG. 2B is a block diagram illustrating the environment contextcorresponding to the reinforcement learning environment, in accordancewith the present disclosure.

FIG. 3 is a block diagram illustrating the multi-processor configurationof the reinforcement learning processor, in accordance with the presentdisclosure.

FIG. 4 is a block diagram illustrating the configuration of a System onChip (SoC) incorporating the reinforcement learning processor, inaccordance with the present disclosure.

FIG. 5 is a block diagram illustrating the configuration of a PrintedCircuit Board (PCB) incorporating the reinforcement learning processor,in accordance with the present disclosure.

FIG. 6A and FIG. 6B in combination illustrate a flow-chart explainingthe steps involved in a method for implementing predeterminedreinforcement learning using the reinforcement learning processor, inaccordance with the present disclosure.

FIG. 7A is a block diagram illustrating a reward function approximator,in accordance with the present disclosure.

FIG. 7B is a block diagram illustrating an exemplary deep neural networkimplementing the reward function approximator described in FIG. 7A.

FIG. 7C is a block diagram illustrating a Generative Adversarial Network(GAN), used for reward function approximation, in accordance with thepresent disclosure.

FIG. 8 illustrates a programmable CISFD generating thread blocks, inaccordance present disclosure.

FIG. 9A to 9E illustrates the plurality of threads blocks gene ed by theCISFD. FIG. 9A illustrates a r-micro thread block, in accordance withthe present disclosure.

FIG. 10A is a block diagram illustrating a training algorithmimplemented by the processor architecture for training reinforcementlearning agents in the present disclosure.

FIG. 10B illustrates a flowchart depicting a method to implementtraining of reinforcement learning agents, in accordance with thepresent disclosure.

Although the specific features of the embodiments herein are shown insome drawings and not in others. This is done for convenience only aseach feature may be combined with any or all of the other features inaccordance with the embodiments herein.

DETAILED DESCRIPTION

In the following detailed description, a reference is made to theaccompanying drawings that form a part hereof, and in which the specificembodiments that may be practiced is shown by way of illustration. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the embodiments and it is to be understood thatother changes may be made without departing from the scope of theembodiments. The following detailed description is therefore not to betaken in a limited sense.

In view of the drawbacks discussed hitherto, there was felt a need for aprocessor specifically designed for and specialized in executingreinforcement learning operations. In order to address theaforementioned need, the present disclosure envisages a processor thathas been specifically configured (programmed) to execute reinforcementlearning operations by the way of implementing an instruction set (anapplication-specific instruction set) which incorporates instructionsspecifically designed for the implementation of reinforcement learningtasks/operations.

The present disclosure envisages a processor (termed as ‘reinforcementlearning processor’ hereafter) specifically configured to implementreinforcement learning tasks: operations. In accordance with the presentdisclosure, the application-specific instruction set executed by thereinforcement learning processor incorporates ‘Single InstructionMultiple Agents (SIMA)’ type instructions. SIMA type instructions arespecifically designed to be implemented simultaneously on a plurality ofreinforcement learning agents which in turn are interacting withcorresponding reinforcement learning environments.

In accordance with the present disclosure, the SIMA type instructionsare specifically configured to receive either a reinforcement learningagent ID or a reinforcement learning environment ID as the operand. Thereinforcement learning agent ID (RL agent ID) corresponds to areinforcement learning agent, while the reinforcement learningenvironment ID (RL environment ID) corresponds to a reinforcementlearning environment (with which the reinforcement learning agentrepresented by reinforcement learning agent ID interacts). The SIMA typeinstructions envisaged by the present disclosure, when executed by thereinforcement learning processor perform predetermined reinforcementlearning activities directed onto either a reinforcement learning agentor a corresponding reinforcement learning environment specified as apart (operand) of the SIMA type instructions.

In accordance with an exemplary embodiment of the present disclosure,the SIMA type instructions when executed by the reinforcement processor,trigger a reinforcement learning agent to interact with a correspondingreinforcement learning environment and further enable the reinforcementlearning agent to explore the reinforcement learning environment anddeduce relevant learnings from the reinforcement learning environment.Additionally, SIMA type instructions also provide for the deducedlearnings to be iteratively applied onto the reinforcement learningenvironment to deduce furthermore learnings therefrom.

Further, the SIMA type instructions when executed by the reinforcementlearning processor, also enable the reinforcement learning agent toexploit the learnings deduced from any previous interactions between thereinforcement learning agent and the reinforcement learning environment.Further, the SIMA type instructions also enable the reinforcementlearning agent to iteratively exploit the learnings deduced from theprevious interactions, in any of the subsequent interactions with thereinforcement learning environment. Further, the SIMA type instructionsalso provide for construction of a Markov Decision Process (MDP) and aSemi-Markov Decision Process (SMDP) based on the interaction between thereinforcement learning agent and the corresponding reinforcementlearning environment.

Further, the SIMA type instructions also enable selective updating ofthe MDP and SMDP, based on the interactions between the reinforcementlearning agent and the corresponding reinforcement learning environment.The SIMA type instructions, when executed by the reinforcement learningprocessor, also backup the MDP and SMDP. Further, the SIMA typeinstructions when executed on the reinforcement learning agent, enablethe reinforcement learning agent to initiate a Q-learning procedure, anda deep-learning procedure and also to associate a reward function inreturn for the Q-learning and the deep-learning performed by thereinforcement learning agent.

Further, the SIMA type instructions, upon execution by the reinforcementlearning processor, read and analyze the ‘learning context’corresponding to the reinforcement learning agent and the reinforcementlearning environment. Further, the SIMA type instructions determine anoptimal Q-value corresponding to a current state of the reinforcementlearning agent, and trigger the reinforcement learning agent to performgeneralized policy iteration, and on-policy and off-policy learningmethods. Further, the SIMA type instructions, upon execution,approximate a state-value function and a reward function for the currentstate of the reinforcement learning agent. Further, the SIMA typeinstructions, when executed by the reinforcement learning processor,train at least one of a deep neural network (DNN) and a recurrent neuralnetwork (RNN) using a predetermined learning context, and furthertrigger the deep neural network or the recurrent neural network forapproximating at least one of a reward function and state-value functioncorresponding to the current state of the reinforcement learning agent.

Referring to FIG. 1, there is shown a block diagram illustrating thecomponents of the system 100 for implementing the tasks/operationspertinent to reinforcement learning. The system 100, as shown in FIG. 1includes a first memory module 10 (preferably an IRAM). The first memorymodule stores the application-specific instruction set (ASI), whichincorporates the SIMA instructions (referred to as ‘instructions’hereafter) for performing predetermined reinforcement learning tasks.The instructions, as described in the above paragraphs, are configuredto incorporate either a reinforcement learning agent ID or areinforcement learning environment ID as the operand. The reinforcementlearning agent ID represents a reinforcement learning agent (not shownin figures) trying to achieve a predetermined goal in an optimal mannerby the way of interacting with a reinforcement learning environment(represented by reinforcement learning environment ID). Each of theinstructions stored in the first memory module 10 are linked tocorresponding ‘opcodes’. The ‘opcodes’ cot responding to each of theinstructions are also stored in the first memory module 10. Further, thefirst memory module 10 also stores the reinforcement learning agent IDand reinforcement learning environment ID corresponding to each of thereinforcement learning agents and the reinforcement learningenvironments upon which the instructions (of the application-specificinstruction set) are to be implemented.

The system 100 further includes a Complex Instruction Fetch and Decode(CISFD) unit 12 communicably coupled to the first memory module 10. TheCISFD unit 12 fetches from the first memory unit 10, an instruction tobe applied to a reinforcement learning agent or a reinforcement learningenvironment. Subsequently, the CISFD retrieves the ‘opcode’corresponding to the fetched instruction, from the first memory module10. As explained earlier, the instruction fetched by the CISFD unit 12incorporates at least one of a reinforcement learning agent ID and areinforcement learning environment ID as the operand. Depending upon thevalue of the operand, the CISFD unit 12 determines the reinforcementlearning agent/reinforcement learning environment on which the fetchedinstruction is to be implemented.

Subsequently, the CISFD unit 12, based on the ‘opcode’ and ‘operand’corresponding to the fetched instruction, generates a plurality ofpredetermined threads, namely a ‘v-thread’, ‘a-thread’, ‘q-thread’ andan ‘r-thread’, corresponding to the fetched instruction. The threadsgenerated by the CISFD unit 12 are representative of the characteristicsof either the reinforcement learning agent or the reinforcement learningenvironment or both, upon which the fetched instruction is executed. Thecharacteristics represented by the predetermined threads include atleast the action(s) performed by the reinforcement learning agent atevery state, value associated with each state of the reinforcementlearning agent, reward(s) gained by the reinforcement learning agentduring the interaction with the reinforcement learning environment. Inorder to associate each of the threads with the correspondingreinforcement learning agent/reinforcement learning environment, theoperand of the instruction (the instruction for which the threads arecreated) is embedded into the v-thread, a-thread, q-thread and r-thread.

In accordance with the present disclosure, the ‘v-thread’ upon executiondetermines the ‘state-value functions’ corresponding to each state ofthe reinforcement learning agent. The ‘state-value functions’ indicatethe ‘value’ associated with each of the states of the reinforcementlearning agent. Similarly, the ‘a-thread’ upon execution determines the‘actions’ performed by the reinforcement learning agent in every statethereof, and subsequently generates ‘control signals’, for implementingthe ‘actions’ associated with the reinforcement learning agent.Similarly, the ‘q-thread’ upon execution determines ‘Q-values’ which aregenerated using a state-action function representing the actionsperformed by the reinforcement learning agent at every correspondingstate. Similarly, the ‘r-thread’ on execution determines the rewardsobtained by the reinforcement learning agent for performing a specificaction while being in a specific state.

In accordance with the present disclosure, the system 100 furtherincludes a second processor 14 (referred to as ‘reinforcement learningprocessor’ hereafter) specifically configured for executing theinstructions embodied in the application-specific instruction set (ASI),and for implementing the reinforcement tasks represented by theinstructions. The reinforcement learning, processor 14 executes theinstruction fetched by the CISFD unit 12, by the way of executing thecorresponding v-thread, a-thread, q-thread and r-thread. Thereinforcement learning processor 14 is preferably a multi-core processorcomprising a plurality of processor cores.

In accordance with the present disclosure, each of the processor coresof the reinforcement learning processor 14 incorporate at least ‘four’execution units (FIG. 1A describes a processor core 140 having ‘four’execution units 140A, 140B, 140C and 140D). The threads, i.e., v-thread,a-thread, q-thread and r-thread are preferably assigned to individualexecution units of a processor core respectively, thereby causing thethreads (v-thread, a-thread, q-thread and r-thread) to be executed inparallel (simultaneously). The reinforcement learning processor 14,based on the operand associated with the fetched instruction, determinesthe reinforcement learning agent or the reinforcement learningenvironment upon which the threads (i.e., v-thread, a-thread, q-thread,and r-thread) are to be executed. In an example, the reinforcementlearning processor 14 executes the v-thread, a-thread, q-thread andr-thread on a reinforcement learning agent identified by correspondingreinforcement learning agent ID, and determines at least one‘state-value function’, at least one ‘action’, at least one ‘Q-value’,and at least one ‘rewards corresponding to the reinforcement learningagent identified by the reinforcement learning agent ID.

The ‘state-value function’, ‘action’, ‘Q-value’ and ‘reward’ thusdetermined by the reinforcement learning processor 14 are stored in asecond memory module 16. In accordance with the present disclosure, thesecond memory module 16 is preferably bifurcated into at least ‘four’memory partitions, namely, an ‘a-memory module’ 16A, a ‘v-memory module’16B, a ‘q-memory module’ 16C, and an ‘r-memory module’ 16D. The‘a-memory module’ 16A stores the information corresponding to the actionperformed by the reinforcement learning agent (identified by thereinforcement learning agent ID) at every state. The actions are storedon the ‘a-memory module’ 16A in a binary encoded format.

The ‘v-memory module’ 16B stores the ‘state-value functions’ indicativeof the value associated with every state of the reinforcement learningagent (identified by the reinforcement learning agent ID) while thereinforcement learning agent follows a predetermined policy. The‘v-memory module’ 16B also stores the ‘optimal state-value functions’indicative of an optimal state-value associated with the reinforcementlearning agent under an optimal policy. Further, the ‘q-memory module’16C stores ‘Q-values’ which are generated using a state-action functionrepresentative of a correlation between the actions performed by thereinforcement learning agent at every state and under a predeterminedpolicy. The ‘q-memory module’ 16C also stores tale ‘optimal Q-value’ forevery state-action pair associated with the reinforcement learningagent, and adhering to an optimal policy. The term ‘state-actionfunction’ denotes the action performed by the reinforcement learningagent at a specific state. Further, the ‘r-memory module’ 16D stores the‘rewards’ (reward values) obtained by the reinforcement learning agent,in return for performing a specific action while being in a specificstate.

Subsequently, the reinforcement learning processor 14 selectivelyretrieves the ‘state-value functions’, ‘actions’, ‘Q-values’ and‘rewards’ corresponding to the reinforcement learning agent (andindicative of the interaction between the reinforcement learning agentand the reinforcement learning environment) from the ‘a-memory module’16A, ‘v-memory module’ 16B, ‘q-memory module’ 16C, and ‘r-memory module’16D respectively, and transmits the retrieved ‘state-value functions’,‘actions’, ‘Q-values’ and ‘rewards’ to a neural network (illustrated inFIG. 7A) via a corresponding neural network data path 18. Subsequently,the reinforcement learning processor 14 trains the neural network toapproximate reward functions that in turn associate a probable rewardwith the current state of the reinforcement learning agent, and alsowith the probable future states and future actions of the reinforcementlearning agent. Further, the reinforcement learning processor 14 alsotrains the neural network to approximate state-value functions that inturn approximate a probable value for all the probable future states ofthe reinforcement learning agent.

In accordance with an embodiment of the present disclosure, thereinforcement learning processor is further configured to train thereinforcement learning agents using application-domain specificinstruction set (ASI), wherein the application-domain specificinstruction set are single instruction multiple agents (SIMA) basedinstructions, and wherein a plurality of agents are configured to workon a copy of same environment and work on mutually different policies inparallel, and wherein the plurality of agents is selected and merged ina synchronous or asynchronous mann

In accordance with the present disclosure, the CISFD unit 12 isconfigured to receive the SIMA type instructions fetched from the firstmemory module 10 and identify the ‘opcode’ corresponding to the receivedinstruction. Subsequently, the CISFD unit 12 determines and analyzes the‘operand’ (either the reinforcement learning agent ID or thereinforcement learning environment ID) and identifies the correspondingreinforcement learning agent or the reinforcement learning environmentupon which the instruction is to be executed. Subsequently, the CISFDunit 12 converts the instruction into ‘a-thread’, ‘v-thread’, ‘q-thread’and ‘r-thread’ (collectively referred to as a ‘thread block’). The CISFDunit 12 also embeds the corresponding reinforcement learning agent ID orthe reinforcement learning environment ID, so as to associate theinstruction (received from the first memory module 10) with thecorresponding thread block and the corresponding reinforcement learningagent/reinforcement learning environment. Subsequently, each of thethreads, i.e., the ‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’ areassigned to respective execution units of a processor core of thereinforcement learning processor 14. In this case, each of the threadsare simultaneously executed by ‘four’ execution units of the processorcore.

In accordance with an exemplary embodiment of the present disclosure, ifthe CISFD unit 12 fetches the instruction ‘optval agent ID’, then theCISFD unit 12 decodes the instruction to determine the ‘opcode’corresponding to the instruction, and subsequently determines thefunction to be performed in response to the instruction, based on the‘opcodes’. Subsequently, the CISFD unit 12 triggers the creation of the‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’ corresponding to theinstruction ‘optval’, and triggers the reinforcement learning processor14 to execute the ‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’ aspredetermined. Arithmetic Logic Unit (ALU) operations. The CISFD unit 12instructs the reinforcement learning processor 14 to execute the‘a-thread’, ‘v-thread’ ‘q-thread’ and ‘r-thread’ on the reinforcementlearning agent/reinforcement learning environment identified by the‘operand’ (reinforcement learning agent ID/reinforcement learningenvironment ID). The resultant of the execution of the threads arestored in ‘a-memory module’ 16A, ‘v-memory module’ 16B, ‘q-memorymodule’ 16C, and ‘r-memory module’ 16B respectively.

In accordance with the present disclosure, during the execution of the‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’ by the reinforcementlearning processor 14, if the instruction corresponding to theaforementioned threads includes a reinforcement learning agent ID as anoperand, then in such a case, the reinforcement learning processor 14accesses the ‘agent context’ (described in FIG. 2A) corresponding to thereinforcement learning agent identified by the reinforcement learningagent ID. Subsequently, the reinforcement learning processor 14, by theway of executing the predetermined operations (on the context registerstoring the ‘agent context’) determines the states associated with thereinforcement learning agent, actions to be performed by thereinforcement learning agent, rewards accrued by the reinforcementlearning agent, and the policy to be followed by the reinforcementlearning agent. By using the information corresponding to the ‘states’,‘actions’, ‘rewards’ and ‘policy’ associated with the reinforcementlearning agent, the reinforcement learning processor 14 determines‘state-value functions’, ‘actions’, ‘Q-values’ and ‘rewards’corresponding to the reinforcement learning agent. Subsequently, the‘state-value functions’, ‘actions’, ‘Q-values’ and ‘rewards’ aretransmitted by the reinforcement learning processor 14 to the secondmemory module 16 for storage.

In accordance with the present disclosure, the CISFD unit 12 could beconceptualized either as a fixed hardware implementation or as aprogrammable thread generator. In the event that the CISFD unit 12 isconceptualized as a fixed hardware implementation, then each, of theinstructions is decoded and subsequently executed by dedicated hardware.Alternatively, if the CISFD unit 12 is conceptualized as a programmablethread generator, then each instruction is mapped to output threads(‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’). The output threadsare preferably sequence of ‘Read Modify Write (RMW)’ operationsperformed on respective memory modules (‘a-memory module’ 16A, ‘v-memorymodule’ 16B, ‘q-memory module’ 16C, and ‘r-memory module’ 16D), with the‘Modify (M)’ operation being performed as an ALU operation.

In accordance with the present disclosure, each of the processor cores(a processor core 140 illustrated in FIG. 1A) of the reinforcementlearning processor 14 incorporate predetermined number of executionunits (execution units 140A, 140B, 140C and 140D illustrated in FIG.1A). Each of the execution units execute the threads corresponding tothe SIMA instruction fetched from the first memory module 10. As shownin FIG. 1B, an execution unit 140A incorporates a fetch/decode unit 14Aand a dispatch/collection unit 14B. The fetch/decode unit 14A isconfigured to fetch the ALU instructions from the CISFD unit 12 forexecution. The dispatch/collection unit 14B accesses the ‘a-memorymodule’ 16A, ‘v-memory module’ 16B, ‘q-memory module’ 16C, and ‘r-memorymodule’ 16D to complete the execution of the threads (‘a-thread’,‘v-thread’, ‘g-thread’ and ‘r-thread’). Further, the execution unit 140Aalso stores the ‘learning context’ and the ‘inferencing context’corresponding to the reinforcement learning agent/reinforcement learningenvironment represented by the operand of the SIMA instruction. The‘learning context’ and the ‘inferencing context’ are stored across aplurality of status registers, constant registers and configurationregisters (not shown in figures).

The term ‘learning context’ represents the characteristics associatedwith a reinforcement learning environment with which the reinforcementlearning agent interacts, and learns from. Further, the term ‘learningcontext’ also represents a series of observations and actions which thereinforcement learning agent has obtained as a result of the interactionwith the reinforcement learning environment. The term ‘inferencingcontext’ represents the manner in which the reinforcement learning agentbehaves (i.e., performs actions) subsequent to learning from theinteraction with the reinforcement learning environment.

In accordance with the present, disclosure, execution of each of theSIMA instructions is denoted using a ‘coreID’. The ‘coreID’ isdetermined based on the processor core executing the SIMA instruction.Further, each of the learning contexts stored in the correspondingexecution unit of the processor core (executing the SIMA instruction)are identified using a ‘contextID’. Further, each of the threads (i.e.,‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’) corresponding to theSIMA instruction are identified by a combination of a ‘threadID’ and the‘coreID’ and the ‘contextID’. The combination of ‘threadID’, ‘coreID’and the ‘contextID’ is illustrated in FIG. 1C.

In accordance with the present disclosure, ‘learning context’ isassociated with the ‘exploration’ activities performed by a particularreinforcement learning agent within a corresponding reinforcementlearning environment. The term ‘exploration’ characterizes theactivities in which the reinforcement learning agent interacts with thereinforcement learning environment by the way of performingpredetermined actions, and learns from the interactions with thereinforcement learning environment. Further, ‘inferencing context’ isassociated with the ‘exploitation’ activities performed by thereinforcement learning agent within the corresponding reinforcementlearning environment. The term ‘exploitation’ characterizes theactivities in which the reinforcement learning agent decides upon theactions to be performed, based on the learnings gathered from theprevious interactions with the reinforcement learning environment. Anyof the well known algorithms such as epsilon-greedy algorithm, Boltzmannsweep algorithm, epsilon-soft policy, softmax action selection policyare utilized for the purpose of ‘exploitation’, depending upon thesuitability of the algorithm to the reinforcement learning environment.

Referring to the block diagram illustrated in FIG. 1D, the term‘agentID’ denotes the reinforcement learning agent performing eitherexploitation of exploration activity. The term ‘envID’ denotes thereinforcement learning environment within which the reinforcementlearning agent is performing either the exploration or exploitationactivity. A particular reinforcement learning environment (denoted by‘envID’) could interact with multiple reinforcement learning agents(denoted by respective agentIDs). Therefore, ‘agentID’ represents thespecific reinforcement learning agent interacting with the reinforcementlearning environment. Further, ‘coreID’ represents the processor core(of the reinforcement learning processor 14) executing the SIMAinstruction applied upon either the reinforcement learning agentidentified by ‘agentID’ or the reinforcement learning environmentrepresented by ‘envID’. Further, ‘contextID’ represents the context(exploitation or exploration) of the activity performed by thereinforcement learning agent (represented by ‘agentID’) by the way ofinteraction with the reinforcement learning environment (represented by‘envID’), as a part of the execution of the SIMA instruction. Everyprocessor core (represented by ‘coreID’) is configured to simultaneouslyimplement multiple contexts (represented by respective contextIDs).Further, ‘threadID’ represents the threads (i.e., ‘a-thread’,‘v-thread’, ‘q-thread’ and ‘r-thread’) corresponding to the SIMAinstruction applied onto either the reinforcement learning agentidentified by ‘agentID’ or the reinforcement learning environmentrepresented by ‘envID’.

Referring again to FIG. 1, the system 100 further includes a (scalable)scheduler module 20 that provides the second processor 14 with selectiveaccess to the neural network data paths 18. The scheduler module 20 alsocontrols the operations of a Digital Signal Processing (DSP) unit 26, aMemory Management Unit (MMU) 28, and the software driver modules(‘Hostif’ drivers) 30 that facilitates communication between thereinforcement learning processor 14 and a first processor 24 (referredto as ‘host processor’ hereafter). Further, the scheduler module 20allocates memory space in each of the aforementioned memory modules(i.e., ‘a-memory module’ 16A ‘v-memory module’ 16B, ‘q-memory module’16C, and ‘r-memory module’ 16D) for the results of the execution of the‘a-thread’, ‘v-thread’, ‘q-thread’ and ‘r-thread’ to be written thereto.Additionally, the scheduler module 20 is also configured to selectivelyprioritize the activities (exploration and exploitation) ofpredetermined reinforcement learning agents. Further, the schedulermodule 20 also prioritizes the activities (exploration and exploitation)performed upon predetermined reinforcement learning environments.Additionally, the scheduler module 20 selectively prioritizes theinteraction between the reinforcement learning processor 14 and the hostprocessor 24.

Referring to FIG. 1 again, the ‘a-memory module’ 16A stores informationcorresponding, to the actions performed by the reinforcement learningagent. The ‘v-memory module’ 16B stores the ‘state-value functions’indicative of the value associated with every state of the reinforcementlearning agent, under a predetermined policy. The ‘v-memory module’ 16Balso stores the ‘optimal state-value function’ indicative of an optimalvalue associated with every state of the reinforcement learning agentunder an optimal policy. Further, the ‘q-memory module’ 16C stores‘Q-values’ which are generated using a state-action function whichrepresents a correlation between the actions performed by thereinforcement learning agent at every state and under a predeterminedpolicy. The ‘q-memory module’ 16C also stores the ‘optimal Q-values’ forevery state-action pair associated with the reinforcement learningagent, and adhering to an optimal policy. The term ‘state-action pair’denotes the action performed by the reinforcement learning agent at aspecific state. Further, the ‘r-memory module’ 16D stores the ‘rewards’(reward values) obtained by the reinforcement learning agent, in returnfor performing a specific action while being in a specific state.

In accordance with the present disclosure, the ‘v-memory module’ 16B isbifurcated into a plurality of memory banks as shown in FIG. 1E, witheach memory bank of the ‘v-memory module’ 16B storing ‘state-valuefunction Vπ(S)’ indicative of the value of a specific state of thereinforcement learning agent under a predetermined policy. As shown inFIG. 1E, the ‘v-memory module’ 16B is bifurcated into ‘four’ exemplarymemory banks 160B, 161B, 162B and 163B. For example, memory bank 160Bstores the ‘state-value function’ indicative of a ‘first’ valuecorresponding to a ‘first’ state of the reinforcement learning agentunder a predetermined policy. Further, memory bank 161B stores the‘state-value function’ indicative of a ‘second’ value corresponding to a‘second’ state of the reinforcement learning agent under thepredetermined policy. Further, memory bank 162B stores the ‘state-valuefunction’ indicative of a ‘third’ value corresponding to a ‘third’ stateof the reinforcement learning agent under the predetermined policy.Further, memory bank 163B stores the ‘state-value function’ indicativeof a ‘forth’ value corresponding to a ‘forth’ state of the reinforcementlearning agent under the predetermined policy.

In accordance with the present disclosure, the ‘q-memory module’ 16C isbifurcated into a plurality of memory banks as shown in FIG. 1F. Asshown in FIG. 1F, the ‘q-memory module’ 16A is bifurcated into ‘four’exemplary memory banks 160C, 161C, 162C and 163C. Each memory bank ofthe ‘q-memory module’ 16C stores the ‘Q-value’ corresponding to anaction performed by the reinforcement learning agent, at a particularstate and under the predetermined policy. For example, memory bank 160Cstores the ‘Q-value’ corresponding to a ‘first’ action performed by thereinforcement agent while being at a ‘first’ state and following apredetermined policy. Further, memory bank 161C stores the ‘Q-value’corresponding to a ‘second’ action performed by the reinforcement agentwhile being at a ‘second’ state and following the predetermined policy.Further, memory bank 162C stores the ‘Q-value’ corresponding to a‘third’ action performed by the reinforcement agent while being at a‘third’ state and following the predetermined policy. Further, memorybank 163C stores the ‘Q-value’ corresponding to a ‘forth’ actionperformed by the reinforcement assent while being at a ‘forth’ state andfollowing the predetermined policy.

Similarly, the ‘r-memory module’ 16D is bifurcated into a plurality ofmemory banks as shown in FIG. 1G, with each memory bank of the ‘r-memorymodule’ 16D storing a ‘reward value’ indicative of the reward obtainedby the reinforcement learning agent, in return for performing a specificaction (while being) in a specific state, and under the predeterminedpolicy. As shown in FIG. 1G, the ‘r-memory module’ 16D is bifurcatedinto ‘four’ exemplary memory banks 160D, 161D, 162D and 163D. The memorybank 160D stores a reward value indicative of the reward obtained by thereinforcement learning agent, in return for performing a ‘first’ action(while being) in a ‘first’ state, under a predetermined policy. Further,memory bank 161D stores a reward value indicative of the reward obtainedby the reinforcement learning agent, in return for performing a ‘second’action in a ‘second’ state under the predetermined policy. Further,memory bank 162D stores a reward value indicative of the reward obtainedby the reinforcement learning agent, in return for performing a ‘third’action in a ‘third’ state under the predetermined policy. Further,memory bank 163D gores a reward value indicative of the reward obtainedby the reinforcement learning agent, in return for performing a ‘forth’action in a ‘forth’ state under the predetermined policy.

In accordance with the present disclosure, the ‘v-memory module’ 16B,the ‘q-memory module’ 16C and the ‘r-memory module 16D are accessed bythe CISFD unit 12 and the reinforcement learning processor 14 by using acombinational sequence of ‘envID’, ‘agentID’, ‘stateID’, and ‘actionID’.As explained in the aforementioned paragraphs, the term ‘envID’ denotesthe reinforcement learning environment within which the reinforcementlearning agent is performing either the exploration or exploitationactivity, and the term ‘agentID’ denotes the reinforcement learningagent performing either the exploration or exploitation activity.Further, ‘stateID’ denotes the current state of the reinforcementlearning agent, and the term ‘actionID ’ denotes the action undertaken(performed) by the reinforcement learning agent while being in aspecific state which is denoted by ‘stateID’.

In the case of ‘v-memory module’ 16B which is bifurcated into memorybanks 160B, 161B, 162B and 163B, memory bank 160B stores the‘state-value function’ indicative of a ‘first’ value corresponding to a‘first’ state (represented by stateID) of the reinforcement learningagent (represented by agentID) under the predetermined policy. The‘envID’ representing the reinforcement learning environment remainsunchanged for the ‘four’ exemplary memory banks 160B, 161B, 162B and163B since the reinforcement learning agent (represented by ‘agentID’)interacts with only one reinforcement learning environment at a givenpoint of time. Similarly, the memory bank 161B stores the ‘state-valuefunction’ indicative of a ‘second’ value corresponding to a ‘second’state (represented by ‘stateID’) of a reinforcement learning agent(represented by ‘agentID’) under the predetermined policy. Similarly,the memory bank 162B stores the ‘state-value function’ indicative of a‘third’ value corresponding to a ‘third’ state (represented by‘stateID’) of a reinforcement learning agent (represented by ‘agentID’)under the predetermined policy. Similarly, the memory bank 163B storesthe ‘state-value function’ indicative of a ‘forth’ value correspondingto a ‘forth’ state (represented by ‘stateID’) of a reinforcementlearning agent (represented by ‘agentID’) under the predeterminedpolicy.

In the case of ‘q-memory module’ 16C which is bifurcated into memorybanks 160C, 161C, 162C and 163C, memory bank 160C stores the ‘Q-value’corresponding to a ‘first’ action (represented by ‘actionID’) performedby the reinforcement agent (represented by ‘agentID’) while being at a‘first’ state (represented by ‘stateID’) and following the predeterminedpolicy. Similarly, memory bank 161C stores the ‘Q-value’ correspondingto a ‘second’ action (represented by ‘actionID’) performed by thereinforcement agent (represented by the ‘agentID’) while being; at a‘second’ state (represented by ‘stateID’) and following thepredetermined policy. Similarly, memory bank 162C stores the ‘Q-value’corresponding to a ‘third’ action (represented by ‘actionID’) performedby the reinforcement agent (represented by the ‘agentID’) while being ata ‘third’ state (represented by ‘stateID’) and following thepredetermined policy. Similarly, memory bank 163C stores the ‘Q-value’corresponding to a ‘forth’ action (represented by ‘actionID’) performedby the reinforcement agent (represented by the ‘agentID’) while being ata ‘forth’ state (represented by ‘stateID’) and following thepredetermined policy.

In the case of ‘r-memory module’ 16D which is bifurcated into memorybanks 160D, 161D, 162D and 163D, the memory bank 160D stores a rewardvalue indicative of the reward obtained by the reinforcement learningagent (represented by ‘agentID’), in return for performing a ‘first’action (represented by ‘actionID’) in a ‘first’ state (represented by‘stateID’), under the predetermined policy. The ‘envID’ representing thereinforcement learning environment remains unchanged for the ‘four’exemplary memory banks 160D, 161D, 162D and 163D since the reinforcementlearning agent (represented by ‘agentID’) interacts with only onereinforcement learning environment at a given point of time. Similarly,memory bank 161D stores a reward value indicative of the reward obtainedby the reinforcement learning agent (represented by ‘agentID’), inreturn for performing a ‘second’ action (represented by ‘actionID’) in a‘second’ state (represented by ‘stateID’), wider the predeterminedpolicy. Similarly, memory bank 162D stores a reward value indicative ofthe reward obtained by the reinforcement learning agent (represented by‘agentID’), in return for performing a ‘third’ action (represented by‘actionID’) in a ‘third’ state (represented by ‘stateID’), under thepredetermined policy. Similarly, memory bank 163D stores a reward valueindicative of the reward obtained by the reinforcement learning agent(represented by ‘agentID’), in return for performing a ‘forth’ action(represented by ‘actionID’) in a ‘forth’ state (represented by‘stateID’), under the predetermined policy.

In accordance with the present disclosure, for the interaction betweenthe reinforcement learning agent and the reinforcement learningenvironment, the ‘Q-values’ are segregated based on the correspondingaction and corresponding state, and subsequently stored in the memorybanks 160C, 161C, 162C and 163C. Similarly, the ‘state-value functions’are segregated, based on the value of the reinforcement learning agentat every specific state, and stored on the memory banks 160B, 161B, 162Band 163B. Similarly, the rewards are segregated based on the action, andthe state and the corresponding action, and stored in the memory banks160D, 161D, 162D and 163D.

In order to obtain an optimal Q-value, the ‘Q-values’ stored on thememory banks 160C, 161C, 162C and 163C are extracted serially (i.e.,beginning with the first bank 160C and moving to the last bank 163C),and the highest of the ‘Q-values’ extracted from the memory banks 160C,161C, 162C and 163C is determined and considered as the optimal Q-value.The state-action pair corresponding to the highest of the Q-values isidentified. The optimal Q-value and the corresponding state-action pairare utilized as a benchmark for iteratively approximating the actions,rewards and an optimal policy for any new states of the reinforcementlearning agent.

Similarly, state-value functions stored on the memory banks 160B, 161B,162B and 163B are retrieved serially (i.e., beginning with the firstbank 160B and moving to the last bank 163B), and the highest value ofthe reinforcement learning agent across the various states (in thiscase, the first state, second state, third state and forth state) isidentified, and is considered as the optimal state-value function. Theoptimal state-value function could be utilized to estimate an optimalpolicy that is considered to be better than any other policies availablefor the reinforcement learning agent.

Similarly, rewards stored on the memory banks 160D, 161D, 162D and 163Dare retrieved serially (i.e., beginning with the first bank 160D andmoving to the last bank 163D), and the highest of the retrieved rewardsis determined, and the corresponding state-action pair is identified.The highest of the retrieved rewards is used as a benchmark forapproximating a reward function, and for determining the manner in whichthe reward obtained by the reinforcement learning agent during thecourse of interaction with the reinforcement learning environment couldbe maximized.

In accordance with the present disclosure, the system 100 furtherincludes a first processor 24 (referred to as ‘host processor’hereafter) configured to create at least one reinforcement learningagent and a corresponding reinforcement learning environment, therebyinitiating a computational approach fix implementing reinforcementlearning operations. The first processor 24 creates at least onereinforcement learning agent and a corresponding reinforcement learningenvironment by the way of programming predetermined, common memorymapped configuration registers. The reinforcement learning agent isrepresented by a 1-bit wide ‘agent register’, while the reinforcementlearning environment is represented by a 1-bit wide ‘environmentregister’. The reinforcement learning agent is represented by ‘agentID’,and the reinforcement learning agent is mapped to the reinforcementlearning environment by 32-bit ‘envID’. In accordance with an exemplaryembodiment of the present disclosure, when the host processor writes thevalue ‘1 or any other predetermined signature on to the predeterminedconfiguration registers, the reinforcement learning agent and thereinforcement learning environment are created. Further, the hostprocessor 24 preferably loads the application-specific instruction setcomprising the instructions for implementing predetermined reinforcementlearning operations, onto the first memory module 10. Subsequently, thehost processor 24 triggers the reinforcement learning processor 14 toexecute the instructions corresponding to the reinforcement learningoperations.

In accordance with the present disclosure, each of the reinforcementlearning agents are associated with an ‘agent context’, and each of thereinforcement learning environments are associated with an ‘environmentcontext’. The ‘agent context’ as shown in FIG. 2A is a collection ofconfiguration registers and host registers, which store informationpertaining to the status the reinforcement learning agent, actions to beperformed by the reinforcement learning agent, reward functionassociated with the agent, policy to be followed by the reinforcementlearning agent, and the like. The ‘environment context’ as shown in FIG.2B is also a collection of configuration registers and host registers,which store the information including ‘agentID’ corresponding to each ofthe reinforcement learning agents interacting with the reinforcementlearning environment, and ‘experimentID’ corresponding to thereinforcement learning experiments performed by each of thereinforcement learning agents.

Referring to FIG. 3, there is shown a block diagram illustrating amulti-processor configuration of the reinforcement learning processor14, in accordance with an exemplary embodiment of the presentdisclosure. The multi-processor configuration of the reinforcementlearning processor 14 (all the reinforcement learning processorscollectively represented by reference numeral 14) is conceptualizedusing an inter-core connection of the reinforcement learning processor14. The reinforcement learning processor 14 incorporates ‘4’ ports toform a communication link (connection) with ‘4’ reinforcement learningprocessors as exemplified in FIG. 3. The reinforcement learningprocessor 14 incorporates a scalable architecture which provides for thenumber of processing cores to be scaled in line with the complexity ofreinforcement learning operations/tasks. The scheduler module 20(described in FIG. 1) of the reinforcement learning processor 14 isconfigured to be implemented on a multi-chip mode and provide thereinforcement learning processor 14 with required internal memory accessand external memory access. Further, the scheduler module 20 alsoprovides for intercommunication between the reinforcement learningprocessors assembled using the multi-processor configuration. The firstmemory module 10 (described in FIG. 1) and the second memory module 16(described in FIG. 1) are not hierarchical, and are configured to beaccessed based on the synchronization between the processing cores ofthe reinforcement learning processor 14. Therefore, the first memorymodule 10 and the second memory module 16 provide for a consistentimplementation of reinforcement learning tasks, even on the multi-chipmode. The reinforcement learning processor 14 when implemented in amulti-processor configuration provides for comparatively higher levelsof parallelism as far as the interaction between a plurality ofreinforcement learning agents and corresponding reinforcement learningenvironments is concerned.

Referring to FIG. 4, there is shown a schematic representation of thesystem 100 as a System on Chip (SOC). The SOC architecture asillustrated in FIG. 4 includes a host processor 24, reinforcementlearning processor 14, sensor controllers 42, DRAM controller, and firstmemory module 10 communicably coupled through a Network on Chip (NOC)architecture. The host processor 24 executes a predetermined hostapplication which communicates with the reinforcement learning processor14 through a mailbox interrupt. The reinforcement learning processor 14is preferably programmed via a host memory map of the host processor 24.Further, the reinforcement learning processor 14 includes aslave/configuration interface and a master interface. The reinforcementlearning processor 14 receives sensory inputs from the sensorcontrollers. The reinforcement learning processor 14 converts thesensory inputs into rewards via at least one of reward functionapproximation, supervisory training and exploitation & exploration ofreinforcement learning environment. The reinforcement learning processor14 samples rewards via at least one of on-chip Input/output (I/O),sensor I/Os (for example 12C, 13C, 12S), and sensor specific ADC (analogto digital converter).

In accordance with the present disclosure, the reinforcement learningprocessor 14 communicates with the host processor 24 communicate throughMailbox interrupts. A Time series database (TSD) stores the time seriesof mailbox interrupts generated by the host processor 24 and thereinforcement learning processor 14. TSD typically includes arrays ofnumbers indexed by time. Typically, a mailbox interrupt generated by thehost processor 24 (host interrupt) is mapped to a mailbox interruptgenerated by the reinforcement learning processor 14. Further, all themailbox interrupts generated by the reinforcement learning processor 14are mapped to the mailbox interrupts generated by the host processor 24.Interrupts are either vectored interrupts or nested interrupts. Theinterrupt messages are generated from a dedicated mailbox implementedvia an on-chip S-RAM (static RAM)

The host processor 24 configures the reinforcement learning processor 14to create reinforcement learning environments and reinforcement learningagents. The host processor 24 is configured to access the memory mappedregisters of the reinforcement learning processor 14. The host processor24 is configured to program the first memory module 10 with theapplication-specific instruction set (ASI) comprising instructions forimplementing predetermined reinforcement learning operations. The hostprocessor 24 further programs the memory mapped registers to configureneural networks, recurrent neural networks and the like. The hostprocessor 24 reads the status of each of the memory mapped registers andthe execution contexts stored therein. The reinforcement learningprocessor 14 is powered, collapsed, clock gated and executed at suitableclock frequencies by the host processor 24.

Referring to FIG. 5, there is shown a schematic representation of thereinforcement learning processor 14 integrated on a Printed CircuitBoard (PCB), in accordance with an exemplary embodiment of the presentdisclosure. The host processor 24 and the reinforcement learningprocessor 14 are integrated onto on the printed circuit board (a singleDie). The host processor 24 and the reinforcement learning processor 14interact via a chip to chip connectivity (52). The chip to chipconnectivity is mapped to one of a high-speed SERDES serial interface orto GPIOs (General Purpose I/Os). The GPIOs are also mapped to hostinterrupts (generated byte host processor 24) and RLP interrupts(generated by the reinforcement learning processor).

Referring to FIG. 6A and FIG. 6B in combination, there is shown aflowchart illustrating the steps involved in the computer-implementedmethod for implementing predetermined reinforcement learning operations.At step 600, a first processor (referred to as host processor hereafter)creates at least one reinforcement learning agent and at least onecorresponding reinforcement learning environment. Subsequently, the hostprocessor assigns a reinforcement learning agent ID to the reinforcementlearning agent, and a reinforcement learning environment ID to thereinforcement learning environment. At step 602, an application-domainspecific instruction set (ASI) comprising instructions for implementingpredetermined reinforcement learning operations is created. Further, atstep 604, each of the instructions are configured to incorporate atleast one of the reinforcement learning agent ID and the reinforcementlearning environment ID as an operand.

At step 606, the application-domain specific instruction set comprisinginstructions for implementing the predetermined reinforcement learningoperations is stored in a first memory module. At step 608, at least oneof the instructions is selectively fetched by a complex instructionfetch and decode (CISFD) unit. Subsequently, the CISFD unit decodes theinstruction and generates a plurality of predetermined threadsrepresentative of the decoded instruction. Further at step 608, theplurality of predetermined threads representing the decoded instructionare embedded with at least one of the reinforcement learning agent IDand reinforcement learning environment ID corresponding to the decodedinstruction.

At step 610, each of the predetermined threads is processed in parallelby a second processor having multiple processing cores (referred to asreinforcement learning processor hereafter). During the execution of thethreads, the instruction corresponding to the threads is applied ontoone of the reinforcement learning agent and reinforcement learningenvironment identified by the reinforcement learning agent ID andreinforcement learning environment ID embedded in the predeterminedthreads. The reinforcement learning processor is configured to implementthe instructions simultaneously on a plurality of reinforcement learningagents and a plurality of reinforcement learning environments. At step612, based on the execution of the threads, action(s) associated withthe reinforcement learning agent at every state thereof, at least onestate-value function, at least one Q-value and at least one reward valueare determined by the reinforcement learning processor. At step 614, thedetermined actions, state-value function, Q-value and reward value arestored in predetermined memory partitions of a second memory module.

At step 616, a neural network is triggered via a corresponding neuralnetwork data path, and the determined actions, state-value function,Q-value and reward value are transferred to the neural network foranalysis. Subsequently, the neural network is trained using thedetermined actions, state-value function, Q-value and reward value, toapproximate a value-function and a reward function corresponding toeither a current state or a future state of the reinforcement learningagent. Shown herein below is a table incorporating a non-exhaustive,exemplary list of instructions for performing predeterminedreinforcement learning operations.

TABLE 1 list of instructions for implementing reinforcement learningoperations Instruction Opcode Operand Description vapproxlin 00000010AgentID Perform value function approximation on a reinforcement learningagent represented by ‘agent ID’. extoptk 00001010 AgentID Trigger areinforcement learning agent represented by ‘agentID’ to enter into anexploitation mode optval 00001100 AgentID Optimal value function forcurrent state of MDP of reinforcement learning agent specified by ‘agentID’. explrk 00001101 AgentID Trigger a reinforcement learning agentspecified by ‘agentID’ to enter into an exploration mode. explrall00001110 EnvID Trigger all the reinforcement learning agents interactingwith a reinforcement learning environment represented by ‘envID’, toenter into exploration mode. vapproxgd 00001111 AgentID Approximate avalue function for reinforcement learning agent represented by ‘agentID’. dwnctxk 00010110 CoreID Download learning/inferencing context to aprocessor core identified by coreID. policygradk 00011001 AgentIDPerform Policy gradient on a reinforcement learning agent represented by‘agent ID’. bkpk 00011010 AgentID Perform a ‘backup’ operation an areinforcement learning agent represented by ‘agent ID’. politerk00011101 AgentID Perform Policy iteration on a reinforcement learningagent represented by ‘agent ID’. fetchq 00011111 AgentID Fetch Q-valuesfor reinforcement learning agent represented by ‘agent ID’, from Q-memory module. Q-value for the last state of the reinforcement learningagent is fetched, by the way of a stack based access. fetchv 00100000AgentID Fetch value functions for reinforcement learning agentrepresented by ‘agent ID’, from V-memory module. Value function for thelast state of the reinforcement learning agent is fetched, by the way ofa stack based access. dmar 00100001 DMA length Perform a Direct MemoryAccess (DMA) (16 bit), DMA start on the R-memory module. offset (15bit), direction (1 bit) dmaa 00100010 DMA length Perform a Direct MemoryAccess (DMA) (16 bit), DMA start on the A-memory module. offset (15bit), direction (1 bit) dmav 00100111 DMA length Perform a Direct MemoryAccess (16 bit), DMA start (DMA) on the V-memory module. offset (15bit), direction (1 bit) dmaq 00101000 DMA length (16 bit), Perform aDirect Memory Access DMA start offset (DMA) on the Q-memory module. (15bit), direction (1 bit) ql 00101001 AgentID Initiate Q learning on areinforcement learning agent represented by ‘agent ID’. gpi 00101100AgentID Perform a Generic Policy Iteration (GPI)on a reinforcementlearning agent represented by ‘agent ID’. gpiall 00101101 EnvID Performa Generic Policy Iteration (GPI)on all reinforcement learning agentsinteracting with the reinforcement learning environment represented by‘envID’. modelenv 00101110 AgentID Model the reinforcement learningenvironment for a reinforcement learning agent represented by ‘agentID’. modelenvall 00101111 EnvID Model the reinforcement learningenvironment represented by ‘envID’ for interaction with correspondingreinforcement learning agents. fetchr 00110100 AgentID Fetch rewardvalues for reinforcement learning agent represented by ‘agent ID’, fromR-memory module. Reward value for the last state of the reinforcementlearning agent is fetched, by the way of a stack based access. optq00110101 AgentID Determine the optimal Q-value for the current state ofMDP of a reinforcement learning agent represented by ‘agent ID’. crtenv00110110 EnvID Create a reinforcement learning environment with acorresponding ‘envID’. crtagent 00110111 EnvID Create a reinforcementlearning agent in a reinforcement learning environment specified by‘envID’, and assign a ‘agentID’ to the newly created reinforcementlearning agent. assocrewardk 00110111 AgentID, reward Associate a rewardfunction with a functionID (8 bit) reinforcement learning agentrepresented by ‘agent ID’. intrctenvk 00111000 AgentID, N Trigger areinforcement learning agent represented by ‘agent ID’ to interact withreinforcement learning environment for ‘N’ times and update the Q, V, Aand R memory modules, based on the interaction. rdagentctx 00111001EnvID, AgentID Read the agent context corresponding to a reinforcementlearning agent represented by ‘agent ID’, and store the agent contextinto an external memory module. rdenvctx 00111010 EnvID Read theenvironment context corresponding to a reinforcement learningenvironment represented by ‘envID’, and store the environment contextinto an external memory module.

The instructions exemplified in ‘Table 1’ form a pan of theapplication-domain specific instruction set (ASI) envisaged by thepresent disclosure.

In accordance with the present disclosure, the Q-values are determinedbased on the execution of the q-thread. Further, the actions aredetermined based on the execution of the a-thread. Further, thestate-value functions are determined based on the execution of thev-thread. Further, the reward values are determined based on theexecution of the r-thread.

In accordance with the present disclosure, the Q-values are generatedusing a state-action function representative of each state of thereinforcement learning agent, and an action performed by thereinforcement learning agent in the state. Further, the Q-values arestored on the plurality of memory banks (160B, 161B, 162B and 163B) ofthe q-memory module 16B. Every memory bank of the q-memory module storesa Q-value corresponding to ‘one’ state-action pair representative of aninteraction between the reinforcement learning agent and thereinforcement learning environment. Subsequently, the Q-values areextracted from the respective memory banks, and compared foridentification of the highest Q-value amongst the extracted Q-values.Further, the highest Q-value is considered as an optimal Q-value.

In accordance with the present disclosure, the action-values aredetermined by taking into consideration an action performed by thereinforcement learning agent in a predetermined state, and under apredetermined policy. Further, for determining state-value functions, avalue associated with each state of the reinforcement learning agent isdetermined, and represented as a state-value function. Further, fordetermining the reward values, the rewards obtained by the reinforcementlearning agent for performing a predetermined action in a predeterminedstate are determined, and represented as reward values.

Referring to FIG. 7A, there is shown a reward function approximatordenoted by the reference numeral 32. The reward function approximator 32receives high-dimension sensory input training data (denoted byreference numeral 34) and reward labels (denoted by reference numeral36). High dimension sensory input training data typically includessensory data received from the environment in which the reinforcementlearning processor 14 is installed. For example, if the reinforcementlearning processor 14 is installed in a Printed Circuit Board (PCB),then the sensory data obtained from the environment described by the PCBincludes images, videos, acoustics, audio, speech, temperature,pressure, digitized analog signals and the like. The sensory inputtraining data is considered to be either favorable or non-favorable, andaccordingly corresponding reward is calculated. The reward labelstypically indicate the nature of the reward associated with the sensoryinput training data. For example, the rewards associated with thesensory input data could either be positive or negative or neutral(zero). In accordance with the present disclosure, the rewardapproximator 32 is function configured to be implemented in ‘two’predetermined modes, namely, training mode and inference mode. Whenimplemented in training mode, the reward function approximator 32 istrained to approximate a reward function. When implemented in inferencemode, the reward function approximator 32 provides rewards 38 to thereinforcement learning processor 14. The reinforcement learningprocessor 14 stores the rewards thus obtained in the ‘r-memory module’16D.

Referring to FIG. 7B, there is shown an exemplary deep neural network700 for implementing the reward function approximator described in FIG.7A. Reference numerals 44 and 46 denote high-dimension sensory inputtraining data and reward labels respectively. Reference numeral 48Arepresents a connection between high-dimension sensory input trainingdata 44, reward labels 46 and neurons of the input layer 48 of the deepneural network 700. Reference numeral 50 represents the phase of‘feature extraction’ from the high-dimension sensory input training data44 and reward labels 46. Reference numeral 52 represents multiple hiddenlayers of the deep neural network 700 located in between the input layer48 and the output layer 54. At the output layer 54, error correction(for the features extracted at the input layer 48) is performed. Afterthe aforementioned process is iteratively executed for a predeterminednumber of times, the deep neural network 700 is executed in an inferencemode to generate rewards that are transmitted to the reinforcementlearning processor 14. The reinforcement learning processor 14 storesthe rewards thus obtained in the ‘r-memory module’ 16D.

Under certain circumstances enough data might not be available fortraining a convolutional neural network or a recursive neural network.In such scenarios, a Generative Adversarial Network (GAN) is utilizedfor reward function approximation. As shown in FIG. 7C, a GAN iscommunicably coupled with a first database 56 storing the features ofrewards generated thus far (also referred to as ‘latent reward space’).A reward signal is generated based on the analysis of the databases 56storing the features of rewards generated thus far. The reward signal issubsequently fed to a reward generator 58. The reward generator 58outputs a reward (referred to as ‘generated reward’ hereafter) that isgenerated based on the latent, reward space. Subsequently, adiscriminator 60 compares the generated reward with ‘real rewards’stored in a second database 56A. The discriminator 60 compares thegenerated rewards with the real rewards, and generates a functionrepresentative of an error rate between the generated rewards and realrewards. The error rate thus obtained is fed back to the rewardgenerator 58 for error correction, via a feedback loop (shown in dottedlines in FIG. 7C). The function representing the error rate isiteratively determined until the error rate reduces to a value lesserthan a predetermined threshold (the threshold calculated based on errortype and error quantity), the training of the GAN is temporarilysuspended. Subsequently, the output provided by the GAN is treated asreward signal, and is transmitted to the reinforcement learningprocessor 14. The reinforcement learning processor 14 stores the rewardsrepresented by the reward signal, in the ‘r-memory module’ 16D.

In accordance with the present disclosure, a computer readablenon-transitory storage medium having computer readable instructionsstored for parallelism in reinforcement learning and trainingreinforcement learning agents. The computer readable instructions whenexecuted by a computer processor, cause the computer processor toperform the following steps:

create at least one reinforcement learning agent and at least onecorresponding reinforcement learning environment, and assign areinforcement learning agent ID to the reinforcement learning agent, andassign a reinforcement learning environment ID to the reinforcementlearning environment;

create an application-domain specific instruction set (ASI), theapplication-domain specific instruction set comprising instructions forimplementing the predetermined reinforcement learning operations;

create each of the instructions by incorporating at least one of thereinforcement learning agent ID and the reinforcement learningenvironment ID therein, as an operand;

configure each of the instructions to be simultaneously implemented on aplurality of reinforcement learning agents and a plurality ofreinforcement learning environments;

selectively fetch at least one of the instructions by using a complexinstruction fetch and decode (CISFD) unit, and decode the instruction,and generate a plurality of predetermined threads corresponding todecoded instruction;

embed into the predetermined threads, at least one of the reinforcementlearning agent ID and reinforcement learning environment ID associatedwith corresponding instruction;

process each of the predetermined threads in parallel, and apply thecorresponding instruction onto at least one of a reinforcement learningagent and reinforcement learning environment identified by thereinforcement learning agent ID and reinforcement learning environmentID embedded in the predetermined threads;

train the reinforcement learning agents using the second processor byexecuting application-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions.

In accordance with the present disclosure, the computer readablenon-transitory storage medium having computer readable instructions,when executed by the computer processor, further cause the computerprocessor to:

create environment vectors corresponding to the reinforcement learningenvironment;

create agent vectors corresponding to the plurality of reinforcementlearning agents;

associate the plurality of reinforcement learning agents to at least onereinforcement learning environment, wherein the reinforcement learningagents follows a different exploration policies as compared with thereinforcement learning environment;

initiate a training on the reinforcement learning agent represented bythe reinforcement learning agent ID by using exploration instruction;

generate a plurality of thread vectors and micro threads by the CISFD inresponse to initiation of training, wherein the plurality of threadvectors comprises q-thread vector, a-thread vector, v-thread vector,g-thread vector and r thread vector; and

map a plurality of reinforcement learning operations to the plurality ofthread vectors, wherein each thread vector corresponds to a plurality ofreinforcement learning agents;

update a gradient vector corresponding to at least one of an agentactor, and an agent critic; wherein the agent actor corresponds to theq-thread vector and the agent critic corresponds to the v-thread vector;

update an action vector from the agent actor corresponding to thea-thread;

determine a gradient parameter of policy dθ and a gradient parameter ofvalue function dθv vector corresponding to the g-thread vector; and

update a policy parameter θ and a parameter of value function θv fromdetermined gradient parameters dθ and dθv in a gradient memory of thesecond memory module.

FIG. 8A illustrates a programmable CISFD generating thread blocks. Inaccordance with an embodiment herein, at least one of the instructionsis fetched from IRAM by using a complex instruction fetch and decode(CISFD) unit. The opcode decoder decodes the instruction, and forwardsthe decoded instructions to instruction parser. A plurality ofpredetermined threads are generated by a thread generator lookup RAMcorresponding to decoded instruction. The instruction parser maps thethreads generated with decoded instructions to generate a plurality ofthread blocks of r, v, q, and g threads. Further, the threads areembedded with one of the reinforcement learning agent ID andreinforcement learning environment ID associated with correspondinginstruction. Subsequently, each of the generated threads are executed inparallel.

FIG. 9A to 9E illustrates the plurality of threads blocks generated bythe CISFD. FIG. 9A illustrates a r-micro thread block. The r-microthread or reward thread is a light thread, which is responsible forsimple read modify write. FIG. 9B illustrates a v-micro thread block.The v-micro thread or value thread is a light thread, which isresponsible for simple read modify write. FIG. 9C illustrates a q-microthread block. The q-micro thread is a light thread, which is simple readmodify write. FIG. 9D illustrates a a-micro thread block. The a-microthread block or a-thread is a light thread, which is simple read modifywrite. FIG. 9E illustrates a g-micro, thread block. The g-micro threador general thread includes a major of heavy threads and few lightthreads for executing complex DSP operations and accessing neuralnetwork data path.

FIG. 10A is a block diagram illustrating a training algorithmimplemented by the processor architecture for training reinforcementlearning agents. In accordance with the disclosure, the processorarchitecture is configured for parallelism in reinforcement learningoperations. The architecture is configured to train reinforcementlearning agents using simile instruction multiple agents (SIMA) basedinstructions to cause a plurality of agents following different policiesexecute in copies of same environments. Further, the best of all agentsis selected and merged in a synchronous or asynchronous manner. Also,the processor architecture is designed to reduce training times by aprocessor architecture with multi agents and distributed training.

In accordance with the disclosure, the processor is configured to createenvironment vectors corresponding to the reinforcement learningenvironment. Further, agent vectors (actors and critics) correspondingto the plurality of reinforcement learning agents are created. Theplurality of reinforcement learning agents is associated to at least onereinforcement learning environment. The reinforcement learning agentsfollows a different exploration policies as compared with thereinforcement learning environment. The processor initiates a trainingon the reinforcement learning agent represented by the reinforcementlearning agent ID by using exploration (exploreall) instruction. Theplurality of thread vectors and micro threads are generated by the CISFDin response to initiation of training using exploreall instruction. Theplurality of thread vectors comprises q-thread vector, a-thread vector,v-thread vector, g-thread vector and r thread vector. Further, aplurality of reinforcement learning operations is mapped to theplurality of thread vectors, where each thread vector corresponds to aplurality of reinforcement learning agents. The plurality of threadvectors are executed for a predetermined time for each reinforcementlearning agent. Thereafter, a gradient vector is updated correspondingto at least one of an agent actor, and an agent critic. The agent actorcorresponds to the q-thread vector and the agent critic corresponds tothe v-thread vector. An action vector from the agent actor is updatedcorresponding to the a-thread. Subsequently, a reward vector is obtainedfrom environment corresponding to the r thread vector. The processordetermines a gradient parameter of policy dθ and a gradient parameter ofvalue function dθv vector corresponding to the g-thread vector, thegradient parameter of policy dθ and a gradient parameter of valuefunction dθv is further updated. Further, a policy parameter θ and aparameter of value function θv is updated from determined gradientparameters dθ and dθv in a gradient memory of the second memory module.

With respect to FIG. 10A, the training follows Advantage Actor Critic(A3C) Method. In accordance with the A3C method, the plurality of agentsincluding actor-critic agents are mapped into a plurality of threads.Further, the plurality of threads are pipelined to increase theefficiency in energy utilization and over all training time. Furthersimilar threading with lower granularity can be applied to other RLmethods like asynchronous n-step Q learning, asynchronous one-stepsarsa, TD, TD(λ) and various future RL algorithms which follow framework of reward, value, Q values, States and policies. In A3C algorithm,each of the operations is mapped into micro threads. The micro threadsare highly pipelined and hence increases the throughput of theprocessor.

In accordance with the disclosure, the thread mapping is implemented toBayesian learning, Single-step Q learning, Sarsa, n step learning etc.

FIG. 10B illustrates a flowchart depicting a method to implementtraining of reinforcement learning agents. In accordance with thepresent disclosure, a method for parallelism in reinforcement learningoperations and training reinforcement learning agents include thefollowing steps:

create at least one reinforcement learning agent and at least onecorresponding reinforcement learning environment, and assign areinforcement learning agent ID to said reinforcement learning agent,and assign a reinforcement learning environment ID to said reinforcementlearning environment;

create an application-domain specific instruction set (ASI), saidapplication-domain specific instruction set comprising instructions forimplementing said predetermined reinforcement learning operations;

create each of said instructions by incorporating at least one of saidreinforcement learning agent ID and said reinforcement learningenvironment ID therein as an operand;

configure each of said instructions to be simultaneously implemented ona plurality of reinforcement learning agents and a plurality ofreinforcement learning environments;

selectively fetch at least one of said instructions by using a complexinstruction fetch and decode (CISFD) unit, and decode said instruction,and generate a plurality of predetermined threads corresponding todecoded instruction;

embed into the predetermined threads, at least one of said reinforcementlearning agent ID and reinforcement learning environment ID associatedwith corresponding instruction;

process each of said predetermined threads in parallel, and apply thecorresponding instruction onto at least one of a reinforcement learningagent and reinforcement learning environment identified by thereinforcement learning agent ID and reinforcement learning environmentID embedded in said predetermined threads;

train the reinforcement learning agents using the second processor byexecuting application-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions.

In accordance with the present disclosure, the reinforcement learningagents are trained using the second processor by executingapplication-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions (1100). The method of trainingreinforcement learning agents includes the step of creating environmentvectors corresponding to the reinforcement learning environment (1102).Further, agent vectors (actors and critics) corresponding to theplurality of reinforcement learning agents are created (1104). Theplurality of reinforcement learning agents is associated to at least onereinforcement learning environment (1106). The reinforcement learningagents follows different exploration policies as compared with thereinforcement learning environment. The processor initiates a trainingon the reinforcement learning agent represented by the reinforcementlearning agent ID by using exploration(exploreall) instruction. Theplurality of thread vectors and micro threads are generated by the CISFDin response to initiation of training using exploreall instruction(1108). The plurality of thread vectors comprises q-thread vector,a-thread vector, v-thread vector, g-thread vector and r thread vector.Further, a plurality of reinforcement learning operations is mapped tothe plurality of thread vectors, and wherein each thread vectorcorresponds to a plurality of reinforcement learning agents (1110). Theplurality of thread vectors are executed for a predetermined time foreach reinforcement learning agent (1112). Thereafter, a gradient vectoris updated corresponding to at least one of an agent actor, and an agentcritic (1114). The agent actor corresponds to the q-thread vector andthe agent critic corresponds to the v-thread vector. An action vectorfrom the agent actor is updated corresponding to the a-thread.Subsequently, a reward vector is obtained from environment correspondingto the r thread vector. The processor determines a gradient parameter ofpolicy dθ and a gradient parameter of value function dθv vectorcorresponding to the g-thread vector, the gradient parameter of policydθ and a gradient parameter of value function dθv is further updated.Further, a policy parameter θ and a parameter of value function θv isupdated from determined gradient parameters dθ and dθv in a gradientmemory of the second memory module (1116).

The technical advantages envisaged by the present disclosure include therealization of a processor architecture that enables extraction andcollection of rich actionable data best suited for reinforcementlearning operations. Further, the present disclosure envisages an ofalternative to general purpose processor architecture such as SingleInstruction Multiple Data (SIMD) and Single Instruction Multiple threads(SIMT). The processor architecture envisaged by the present disclosureis tailor made for effective and efficient implementation ofreinforcement learning tasks/operations. The present disclosure alsoenvisages an application domain specific instruction set designed forexecuting tasks/operations pertinent to reinforcement learning. Theapplication domain specific instruction set is simultaneously executedacross multiple reinforcement learning agents and reinforcement learningenvironments (Single Instruction Multiple Agents (SIMA)). Further, theapplication domain specific instruction set is capable of performingvalue function approximation and reward function approximation, by theway of training a neural network. The application domain specificinstruction set and the processor architecture provide for an effectivebalance to be maintained between exploration and exploitation activitiesof a reinforcement learning environment. Further, the present disclosureprovides an effective solution to the ‘curse of dimensionality’typically witnessed in high-dimensional data analysis scenarios. Theapplication domain specific instruction set and the processorarchitecture envisaged by the present disclosure enable parallellearning and effective sharing of learning, amongst a plurality ofreinforcement learning agents. The processor architecture envisaged bythe present disclosure necessitates fewer clock cycles in comparison tothe conventional CPU/GPU, to implement reinforcement learningoperations/tasks. Further, the application domain specific instructionset and the processor architecture envisaged by the present disclosurerender comparatively larger levels of abstraction, during theimplementation of reinforcement learning operations/tasks. The processorarchitecture of the present disclosure comparatively decrease the timerequired to train all RL agents.

The processor architecture incorporating instruction set based on SingleInstruction Multiple Agents (SIMA) is specifically designed forexecuting tasks/operations pertinent to reinforcement learning inparallel. A processor architecture designed for parallelism inreinforcement learning operations is provided. The processorarchitecture is designed to reduce training times by a processorarchitecture with multi agents and distributed training

An application domain specific instruction set is provided and aprocessor architecture is optimized for training the reinforcementlearning agents.

An autonomous systems and server based applications is developed toprovide the multiple agents as operand to the SIMA instructions, forspeeding up training in the applications like gaming.

A training system for AI systems using SIMA (Single Instruction MultipleAgents) ISA based processor, is provided to support both the replymemory and as well as parallel actor-critic agents, thereby supporting anumber of on policy learning methods.

A training system for AI systems is developed to provide each agent withdifferent r, v, q, g and a threads, thereby providing separate threadfor each one of reinforcement learning operations, to effectivelypipeline a number of parallel agents, to decrease training time by orderof magnitude compared to Advantage Actor Critic (A3C) algorithm whereentire agent is mapped to a single thread.

A training system for AI systems is provided to enable multiple agentsto explore different part of the environment thereby using lesserresource than single DQN, and reducing communication overhead.

A training system for AI systems is provided to use fine grain threadsof RL compared to single thread on entire agent, to enable supportinglarge number of agents in a hierarchical task.

A training system for AI systems using SIMA is developed to provide sameinstructions to multiple agents to reduce code density and create highlypipelined operations with large parallelism in training.

A training system for AI systems, is provided to group similar ofdifferent agents into one thread blocks of operations with SIMAinstructions.

A training system for AI systems is provided to create the agents (actorplus critic) and map the threads, instead of mapping entire of algorithminto single thread.

A training system for AI systems, is provided to map each of theoperations into a micro thread and to pipeline these micro threads,instead of mapping entire agent into a single thread, thereby increasingthe throughput of the processor.

A training system is provided for performing thread mapping to Bayesianlearning, Single-step Q learning, Sarsa, n step Q learning etc.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the embodiments herein that others can, byapplying current knowledge, readily modify and/or adapt for variousapplications such as specific embodiments without departing from thegeneric concept, and, therefore, such adaptations and modificationsshould and are intended to be comprehended within the meaning and rangeof equivalents of the disclosed embodiments.

It is to be understood that the phraseology or terminology employedherein is for the purpose of description and not of limitation.Therefore, while the embodiments herein have been described in terms ofpreferred embodiments, those skilled in the art will recognize that theembodiments herein can be practiced with modifications. However, allsuch modifications are deemed to be within the scope of the claims.

What is claimed is:
 1. A computer-implemented system for trainingartificial intelligence based systems, the system comprising: a firstprocessor configured to create at least one reinforcement learning agentand at least one corresponding reinforcement learning environment, thefirst processor further configured to assign a reinforcement learningagent ID to the reinforcement learning agent, and a reinforcementlearning environment ID to the reinforcement learning environment; afirst memory module communicably coupled to the first processor, thefirst memory module configured to store an application-domain specificinstruction set (ASI), the application-domain specific instruction setcomprising instructions for implementing the predetermined reinforcementlearning operations, each of the instructions incorporating at least oneof the reinforcement learning agent ID and the reinforcement learningenvironment ID as an operand; a complex instruction fetch and decode(CISFD) unit communicably coupled to the memory module, the CISFD unitconfigured to decode at least one of the instructions, and generate aplurality of predetermined threads corresponding to decoded instruction,the CISFD unit still further configured to embed into the predeterminedthreads, at least one of the reinforcement learning agent ID andreinforcement learning environment ID associated with the decodedinstruction; a second processor configured to execute the predeterminedreinforcement learning operations, the second processor comprising aplurality of cores, the second processor configured to process each ofthe predetermined threads in parallel using the plurality of cores, andapply a corresponding instruction onto at least one of a reinforcementlearning agent and reinforcement learning environment identified by thereinforcement learning agent ID and reinforcement learning environmentID embedded in the predetermined threads; a second memory modulecooperating with the second processor, wherein the second memory moduleis a gradient memory, and wherein a plurality of gradients in thegradient memory is updated from each one of a plurality of agentsworking in parallel; and) wherein the second processor is furtherconfigured to train the reinforcement learning agents usingapplication-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions, and wherein a plurality ofagents are configured to work on a copy of same environment and work onmutually different policies in parallel, and wherein the plurality ofagents is selected and merged in a synchronous or asynchronous manner.2. The system as claimed in claim 1, wherein the second processor isconfigured to: create environment vectors corresponding to thereinforcement learning environment; create agent vectors correspondingto the plurality of reinforcement learning agents; associate theplurality of reinforcement learning agents to at least one reinforcementlearning environment, wherein the reinforcement learning agents followsa different exploration policies as compared with the reinforcementlearning environment; initiate a training on the reinforcement learningagent represented by the reinforcement learning agent ID by usingexploration instruction; generate a plurality of thread vectors andmicro threads by the CISFD in response to initiation of training,wherein the plurality of thread vectors comprises q-thread vector,a-thread vector, v-thread vector, g-thread vector and r thread vector;and wherein each operation is mapped into micro thread; and map aplurality of reinforcement learning operations to the plurality ofthread vectors, wherein each thread vector corresponds to a plurality ofreinforcement learning agents; update a gradient vector corresponding toat least one of an agent actor, and an agent critic; wherein the agentactor corresponds to the q-thread vector and the agent criticcorresponds to the v-thread vector; update an action vector from theagent actor corresponding to the a-thread; determine a gradientparameter of policy dθ and a gradient parameter of value function dθvvector corresponding to the g-thread vector; update a policy parameter θand a parameter of value function θv from determined gradient parametersdθ and dθv in a gradient memory of the second memory module.
 3. Thesystem as claimed in claim 1, wherein: the plurality of thread vectorsgenerated by the CISFD are indicative of the interaction between thereinforcement learning agent and the reinforcement learning environment,the plurality of thread vectors are executed for a predetermined timefor each reinforcement learning agent.
 4. The system, as claimed inclaim 1, wherein the second processor is further configured to: triggerthe reinforcement learning agent to interact with the reinforcementlearning environment and enable the reinforcement learning agent to:explore the reinforcement learning environment to deduce relevantlearning therefrom; and apply deduced relevant learning onto thereinforcement learning environment; trigger the reinforcement learningagent to exploit gathered learnings by further interacting with thereinforcement learning environment; construct at least one of a MarkovDecision Process (MDP) and Semi-Markov Decision Process (SMDP) based onan interaction between the reinforcement learning agent and thereinforcement learning environment; selectively update the MDP and SMDP,based on learning achieved from the interaction between thereinforcement learning agent and the reinforcement learning environment;selectively create a backup of the MDP representative of the interactionbetween the reinforcement learning agent and the reinforcement learningenvironment; initiate a q-learning procedure on the reinforcementlearning agent; associate a reward function with the reinforcementlearning agent; enable the reinforcement learning agent to interact withthe reinforcement learning environment for a predetermined time period;read a context corresponding to the reinforcement learning agent, andread the context corresponding to the reinforcement learningenvironment, and store the read context; initiate a predetermineddeep-learning operation on the reinforcement learning agent; determinethe optimal Q-value corresponding to a current state of thereinforcement learning agent; implement generalized policy iteration(GPI); implement on-policy and off-policy learning methods on thereinforcement learning agent; approximate a state-value function for thecurrent state of the reinforcement learning agent, and determine anoptimal state-value function for the current state of the reinforcementlearning agent; and fetch Q-values, state-value functions and rewardvalues corresponding to the reinforcement learning agent, from thecorresponding memory partitions of the second memory module.
 5. Thesystem, as claimed in claim 1, wherein the second processor is furtherconfigured to: feed training data into the neural network through theneural network data path, and initiate training of the neural network;download weights corresponding to the neural network; communicate with apredetermined deep neural network; selectively clear the weightsassigned to the neural network; perform value-function approximationusing the deep neural network, for the reinforcement learning agentrepresented by the reinforcement learning agent ID; and train the deepneural network using a predetermined learning context.
 6. The system asclaimed in claim 1, wherein: the instructions are configured to besimultaneously implemented on a plurality of reinforcement learningagents and a plurality of reinforcement learning environments; andwherein the CISFD unit is further configured to create the predeterminedthreads corresponding to each of the plurality of reinforcement learningagents and plurality of reinforcement learning environments, the CISFDunit still further configured to group the predetermined threads into athread block.
 7. The system as claimed in claim 1, wherein: the secondmemory module is partitioned into a gradient memory to store Globalparameters θ, θv obtained from the training.
 8. A method for trainingartificial intelligence based systems, the method comprising followingcomputer-implemented steps: creating at least one reinforcement learningagent and at least one corresponding reinforcement learning environmentusing a first processor, and assigning a reinforcement learning agent IDto the reinforcement learning agent, and assigning a reinforcementlearning environment ID to the reinforcement learning environment;creating an application-domain specific instruction set (ASI), theapplication-domain specific instruction set comprising instructions forimplementing the predetermined reinforcement learning operations;creating each of the instructions by incorporating at least one of thereinforcement learning agent ID and the reinforcement learningenvironment ID therein, as an operand; storing the application-domainspecific instruction set in a first memory module; selectively fetchingat least one of the instructions, using a complex instruction fetch anddecode (CISFD) unit, and decoding the instruction, and generating aplurality of predetermined threads corresponding to decoded instruction;embedding into the predetermined threads, at least one of thereinforcement learning agent ID and reinforcement learning environmentID associated with the decoded instruction; executing each of thepredetermined threads in parallel using a second processor, and applyingthe corresponding instruction onto at least one of a reinforcementlearning agent and reinforcement learning environment identifiedrespectively by the reinforcement learning agent ID and reinforcementlearning environment ID embedded in the predetermined threads, trainingthe reinforcement learning agents using the second processor byexecuting application-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions.
 9. The method as claimed inclaim 8, wherein the step of creating an application-domain specificinstruction set (ASI), further includes the step of incorporating intothe ASI, the instructions for; creating environment vectorscorresponding to the reinforcement learning environment; creating agentvectors corresponding to the plurality of reinforcement learning agents;associating the plurality of reinforcement learning agents to at leastone reinforcement learning environment, wherein the reinforcementlearning agents follows a different exploration policies as comparedwith the reinforcement learning environment; initiating a training onthe reinforcement learning agent represented by the reinforcementlearning agent ID by using exploration instruction; generating aplurality of thread vectors and micro threads by the CISFD in responseto initiation of training, wherein the plurality of thread vectorscomprises q-thread vector, a-thread vector, v-thread vector, g-threadvector and r thread vector; and mapping a plurality of reinforcementlearning operations to the plurality of thread vectors, wherein eachthread vector corresponds to a plurality of reinforcement learningagents; updating a gradient vector corresponding to at least one of anagent actor, and an agent critic; wherein the agent actor corresponds tothe q-thread vector and the agent critic corresponds to the v-threadvector; updating an action vector from the agent actor corresponding tothe a-thread; determining gradient parameter of policy dθ and gradientparameter of value function dθv vector corresponding to the g-threadvector; and updating policy parameter θ and parameter of value functionθv from determined gradient parameters dθ and dθv in a gradient memoryof the second memory module.
 10. The method as claimed in claim 8,wherein: the plurality of thread vectors generated by the CISFD areindicative of the interaction between the reinforcement learning agentand the reinforcement learning environment, the plurality of threadvectors are executed for a predetermined time for each reinforcementlearning agent.
 11. The method as claimed in claim 8, wherein the stepof creating an application-domain specific instruction set (ASI),further includes the step of incorporating into the ASI, theinstructions for: triggering the reinforcement learning agent tointeract with the reinforcement learning environment and enabling thereinforcement learning agent to: explore the reinforcement learningenvironment to deduce relevant learning therefrom; and apply deducedrelevant learning onto the reinforcement learning environment;triggering the reinforcement learning agent to exploit gathered learningby further interacting with the reinforcement learning environment;constructing at least one of a Markov Decision Process (MDP) andSemi-Markov Decision Process (SMDP) based on interaction between thereinforcement learning agent and the reinforcement learning environment;selectively updating the MDP and SMDP, based on learning achieved fromthe interaction between the reinforcement learning agent and thereinforcement learning environment; selectively creating a backup of theMDP representative of the interaction between the reinforcement learningagent and the reinforcement learning environment; initiating aq-learning procedure on the reinforcement learning agent; associating areward function with the reinforcement learning agent; performinggeneralized policy iteration (GPI); implementing on-policy andoff-policy learning methods on the reinforcement learning agent;enabling the reinforcement learning agent to interact with thereinforcement learning environment for a predetermined time period;reading a context corresponding to the reinforcement learning agent, andreading the context corresponding to the reinforcement learningenvironment, and storing the read context; initiating a predetermineddeep-learning operation on the reinforcement learning agent representedby the reinforcement learning agent ID; determining the optimal Q-valuecorresponding to a current state of the reinforcement learning agent;fetching the Q-values, state-value functions, actions and reward valuescorresponding to the reinforcement learning agent, from thecorresponding memory partitions of the second memory module; andapproximating a state-value function for the current state of thereinforcement learning agent, and determining an optimal state-valuefunction for the current state of the reinforcement learning agent. 12.The method as claimed in claim 8, wherein the step of executing each ofthe predetermined threads in parallel using a second processor, andapplying the corresponding instruction onto at least one of areinforcement learning agent and reinforcement learning environmentidentified respectively by the reinforcement learning agent ID andreinforcement learning environment ID embedded in the predeterminedthreads, and determining an action, a state-value function, a Q-valueand a reward value, based on execution of the instruction correspondingto the predetermined threads, further includes the following steps:determining the Q-values based on the execution of a q-thread;determining actions to be performed by the reinforcement learningprocessor, based on the execution of an a-thread; determining thestate-value functions based on the execution of a v-thread; anddetermining the reward values based on the execution of an r-thread. 13.The method as claimed in claim 8, wherein the step of determining anaction based on the execution of an a-thread, further includes the stepof determining the action performed by the reinforcement learning agentin a predetermined state, and under a predetermined policy.
 14. Themethod as claimed in claim 8, wherein the step of determining thestate-value functions based on the execution of a v-thread, furtherincludes the step of determining a value associated with each state ofthe reinforcement learning agent, and representing the value as astate-value function.
 15. The method as claimed in claim 8, wherein thestep of determining the reward values based on the execution of anr-thread, further includes the step of determining rewards obtained bythe reinforcement learning agent for performing a predetermined actionin a predetermined state, within the reinforcement learning environment,and storing the rewards as reward values.
 16. A computer readablenon-transitory storage medium stored with computer readable instructionsand executed by a computer processor, to cause the computer processorto: create at least one reinforcement learning agent and at least onecorresponding reinforcement learning environment, and assign areinforcement learning agent ID to the reinforcement learning agent, andassign a reinforcement learning environment ID to the reinforcementlearning environment; create an application-domain specific instructionset (ASI), the application-domain specific instruction set comprisinginstructions for implementing the predetermined reinforcement learningoperations; create each of the instructions by incorporating at leastone of the reinforcement learning agent ID and the reinforcementlearning environment ID therein, as an operand; configure each of theinstructions to be simultaneously implemented on a plurality ofreinforcement learning agents and a plurality of reinforcement learningenvironments; selectively fetch at least one of the instructions byusing a complex instruction fetch and decode (CISFD) unit, and decodethe instruction, and generate a plurality of predetermined threadscorresponding to decoded instruction; embed into the predeterminedthreads, at least one of the reinforcement learning agent ID andreinforcement learning environment ID associated with correspondinginstruction; process each of the predetermined threads in parallel, andapply the corresponding instruction onto at least one of a reinforcementlearning agent and reinforcement learning environment identified by thereinforcement learning agent ID and reinforcement learning environmentID embedded in the predetermined threads; train the reinforcementlearning agents using the second processor by executingapplication-domain specific instruction set (ASI), wherein theapplication-domain specific instruction set are single instructionmultiple agents (SIMA) based instructions.
 17. The computer readablenon-transitory storage medium as claimed in claim 16, wherein thecomputer readable instructions, when executed by the computer processor,further cause the computer processor to: create environment vectorscorresponding to the reinforcement learning environment; create agentvectors corresponding to the plurality of reinforcement learning agents;associate the plurality of reinforcement learning agents to at least onereinforcement learning environment, wherein the reinforcement learningagents follows a different exploration policies as compared with thereinforcement learning environment; initiate a training on thereinforcement learning agent represented by the reinforcement learningagent ID by using exploration instruction; generate a plurality ofthread vectors and micro threads by the CISFD in response to initiationof training, wherein the plurality of thread vectors comprises q-threadvector, a-thread vector, v-thread vector, g-thread vector and r threadvector; and map a plurality of reinforcement learning operations to theplurality of thread vectors, wherein each thread vector corresponds to aplurality of reinforcement learning agents; update a gradient vectorcorresponding to at least one of an agent actor, and an agent critic;wherein the agent actor corresponds to the q-thread vector and the agentcritic corresponds to the v-thread vector; update an action vector fromthe agent actor corresponding to the a-thread; determine a gradientparameter of policy dθ and a gradient parameter of value function dθvvector corresponding to the g-thread vector; and update a policyparameter θ and a parameter of value function θv from determinedgradient parameters dθ and dθv in a gradient memory of the second memorymodule.
 18. The computer readable non-transitory storage medium asclaimed in claim 16, wherein the plurality of thread vectors generatedby the CISFD are indicative of the interaction between the reinforcementlearning agent and the reinforcement learning environment, the pluralityof thread vectors are executed for a predetermined time for eachreinforcement learning agent.